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ABSTRACT 

Evalaability assessment is a descriptive and analytic 
process intended to produce a reasoned basis for proceeding with an 
evaluation of. use to both manageaent and policymakers. It na.s jointly 
developed by the members of the program evaluation group of the Orban 
Institute between 1968 and- l^^^e,. The approach begins by obtaining 
management's description of the program. The description is then 
analyzed to determine whether it is: complete* tcceptable to the 
policymakers* and a valid representation of "the program as it 
actually exists. Three types of models are used; logic models, 
function models*, and measurement models. The evaiuator determines 
whether management's expectations are plausible* whether its evidence 
is reliable* and whether the program is feasible to develop. Perhaps 
the most comm'^n problem affectincr federal programs is unrealistic 
expectations* most likelv resulting from a management decision which 
is invalid* implausible* or unacceptable to policymakers. A task for 
the evaiuator is to consstruct alternatives wherever possible. The 
final step is to implement evaluability assessment findings. 
(Author/CIM) 
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PREFACE 



This manual was completed under a contract with the Department of Heialth, Education, and 
Welfare Much ot the work and many of the examples are thus taken from the Bureau of Health 
Planning. The concept of evaluability assessment, however, has been in the process of development 
tor at least 5 years. We drew upon that entire history to write this manual. Where appropriate, we have 
included examples which are more completely documented in separate reports from otheragencies. 
Several instances are cited in which the authors, their colleagues, and our work would have benefited 
from evaluability assessment. We believe that this manual represents a good beginning definition of 
what we hope to be a promising approach to management and evaluation of Government programs. 
We hope that it will continue to be refined by other practitioners. 

Although the context for the process described here is Federal, and most references are to Federal 
managers, the concepts which we outline are applicable to any goal-oriented organization responsi* 
ble for-managing public programs. 

Our intent in developing this method is to provide techniques that can be used in making.public 
programs wori^ better than they now do. We have aimed the docu ment at evaluators who ar6 charged 
with inforniing management and policy officials about the effectiveness of their programs. The 
method, however, goes beyond providing information. We intend for it to forge closer and more useful 
links between management and evaluators, hopeful that the net result will be better program 
c management and more effective programs. 
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I. EVALUABILITY ASSESSMENT: 
WHY IS IT NEEDED? WHAT IS IT? 

^^^^ ♦ 



The Problem iind a Solutidi) 



The ritual enactment of legislation which provides for publicly financed programs engages the 
executive and legislative branches of government inacompact-^an agreement on the existence of a 
problem and of a solution to it. Even if the plfiyers believe the solution to be a partial one, they bold 
high hopes for it. Both sides share adreanfeven if only momehtarily, that this program will work to 
produce the benefit^ intended. They assume-that agency and program hianagers will implement their 
solution wisely and manage the resources placed at their disposal to achieve the desired ends. They 
foresee oversight hearings in which objective evidence of perfomriance is provided by*agency Execu- 
tives, enabling legislators then to modify the legislation accordingly. Finally, they hope, the program 
Will be judged by the pul^ic and their elected representatives as a go6d investment, ably managed. 
One small piece of this'^eam is the assumption that program managers can and will convert the 
legislation into a workabld. program, and that they will have the neceslsary information to help them 
direct the program to the dtesired end. / . 

To this end. Congress and the executive branch have increasingly invested in program evaluation 
over the past decade Starting from nearly nothing in the early sixties, investment in evaluation grew 
to around a quarter of a billioti dollars by 1976. Unfortunately, however, the invesjment has not yet 
paid off Program evaluation has not led to successful policies or programs. Instead, it has been 
piarined and implemented in isolation from Federal decisionmaking, and has produced little informa- 
tion of interest and utility to policymakers ano managers. 

If Frxleral policies and programs were always successful, then the limited utility of the evaluation 
i)f t )dij( t woutcJ be a minor irr tant. However not only are few programs demonstrably successful, 
but many pthers are viewed py the public and elected officials as ineffective, wasteful, and even 
ruirmfui Since programs arejusually a response to some perceived need or problem, abandoning 
thom IS not a politically viabl^ solution. T^e on/y solution is to make programs work ar)d be able to 
domonstrcUo that they do. Iiftoeed, program evaluation was created to help improve policies and 
pr(^qrarns by measuring act(/al performance and by sifting through what works well and what does 

rH)t 

Why has practice fallen to short of expectations? Were the expectations held out for evaluation 
linrt^aii^itic*^ Is it true, as many observers feel, that blame for the failures of evaluation should fall on 
Congr^>ss'> For 1 0 years thin literature has called for more precise objective definition of what needs to 
bt* evalu ated to be induced in legislation. Evaluation would then be able to produce the answers so 
dct^porately needed. Th« viewpoint, however, is both incorrect and dangerous. 

Congross doos not hpve and cannot get enough information to set measurable objectives which 
are also rf^alistic . plaunrie. and achievable. It is reasonable only to expect the Legislature to set goals 
at a conceptual levoK/e.. put a man on the moon within a decade**) and then watch to insure that 
program managers a/e translating objectives into realistic and acceptable measurable definitions of 
ronqrossional intent/lt is the program manager who should develop the information required to s4\ 



su^4%jectivos. sine 



the program evaluation function is one important source of that information. 
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Yet as many have noted, the program evaluation.functjons established in government have not 
been used by program managers. The authors of this manual believe that the failure of evaluation to 
help produce effective policies and programs stems from a number of factors relating both to the «ay 
programs and the function of evaluation are designed and managed. 

• • 
In our research over th^ last 5 ydars. we have defined a number of conditions (Horst et al., 1974) 
which. If present, generally disable attempts to evaluate performance. More fundamentally, though, 
these conditions also disable attempts to manage prbgrams for important results of national concern. 
Having identified such roadblocks, we attempted to develop methods to eliminate them. The result of 
that effort is a method we call evaluability assessment (EA). 

Evaluability assessment is an evaluation technique and management decision process designed 
to deal with the recurnng problem of managers ahd policymakers who frequently are unable to guide 
programs toward desired performance and outcomes. Even with substantial evaluation efforts, some 
programs do not tmprove over time. We believe thjs is because of deficiencies in the descriptions of 
programs and the administrative structure of government that hamper maoagement and evalSators. 
With EA. the manager is helped to estabjish conditions that are prerequisite to success. 

Evaluability assessment begins with an analysis of management's description of its program to 
determine whether or not it meets the method criteria. When even one of the questions is answered 
• no. there »s a h'gh probability that the program will not be successful and evaluation information will 
not be useful. The existing program description is said to beunevaluable since no useful evaluation is 
likely 

i- ■ ■ . 

Part of EA requires the evaluator and i.ianagement to work together to remove cc-^ditions that 
make the program description unevaluable. The result of an EA is a sei of manauament decisions 
that establish an evaluable program description. With such a description management can be 
confident that evaluation will show the program to be successful in terms ac -'ptable to policymak- 
e^s and will provide management with the capability to achieve performance. 



What Is Evaluability Assessment? 



Evaluability assessment is a descriptive and analytic process intended to produce a reasoned 
basis for proceeding with an evaluation of use to both management and policymakers. It was jointly 
developed by the members of the program evaluation group of The Urban Institute between 1 968 and 
1 978 While all versions of the method are based on the applied experiences of the members of the 
o'oup. definitions and sequence of operations vary slightly among the different practitioners. Joseph 
S Wholey. in Evaluation: Promise and Performance (in press), describes his approach. 

Basically, the'^roup has taken the problems we have observed over the pas* 6 years and converted 
th*;m mto a sei of criteria which must be satisfied before proceeding to full evaluation. The criteria are 
used to guide the evaluation through a series of planning and analysis tasks designed to identify and 
thefi eliminate roadblocks. 

fho approach basically begins by obtaining management s descnption of the program. The 
M«scription IS then systematically analyzed to determined whether it meets the following require- 
ments 

• It IS complete. ' * 

• It is acceptable to policymakers; 

• It IS ci valid representation of the program as it actually exists; 
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• the expectations for the program are plausible; 



• The evidence' required i>y management can be reliably produced; 

• The evidence required by management is feasible to collect; and 

• Management s intended use of the information can realistically be 

expected to affect performance. » ^ ^ 

Eacfi of these points is a question to be answered by the evaluatoc. working with the manager. We 
also add a function for th« evaluator which is not traditional— ^hen the analysis reveals gaps or 
problems in management s description, he should be involved in devising alternative descriptions 
which will resolve the problems. Jhis function puts the evaluator into both program and management 
system dqsign. based-ton an assumption that if evaluation is to be used by management to affect 
perforinance. it is inseparable from these design needs. 

t 

What Are the Conclusions an Evaluability Assess- 
ment Can Reach? 

The object of an EA is to arrive at a program description that is evaluable. In other words, the 
descnption permits the progr.-.m to be measured with some reasonable assurance that the evaluation 
can be done and that predetermined expectations can be realized. As noted previously, if even one 
criterion is not met. the program is judged to be unevaluable. meaning that there is a high risk that 
management will not be able to demonstrate or achieve program success in terms acceptable to 
policymakers \ 

There are three types ot unevaluable conclusions that can be reached: 

1 Comparison of program management's description with the lis^of prescribed elements show 
the description is incomplete. Certain parts of the description are not available, indicating that 
management has not yet made some key decisions. For example, ^n objective may not have been 
defined in terms of measures and expectations. Usually there are people with ideas about how to 
define an obiective.- but the management decision on definition may be missing. An incomplete 
description can be easily iixed by management. 

2 Gomparison of the evaluator's and operator's description with program management's de- 
scnpnon indicates th^ program management's descnption is implausible, invalid, not cost-feasible 
t&-^valuate. or not useful Problems raised by this comparison rnean that either management's 
description is inadequate or the evaluator s description is wrong. The evaluator will present evidence 
to show why management s description is unevaluable. If management disagrees, the evaluator then 
requires additional inforination on the program activity to resolve the disagreement. 

3 Comparison of pohcymakers' and program managements description shows that program 
management s descnption is unacceptable If policymakers disagree with program management, it 
IS generally because program management has either omitted an objective or set of activities 
important to policymakers or included^objectives or activities with which policymakers disagree. What 
ts needed in such cases is a process^whereby disagreements can be aired and validated and missing 
elements added through a program design activity introduced by management. It may be possible to 
change policymaker descriptions after analysis of the areas of disagreement. 

An evaluable conclusion, on the other hand, produces a set of decisions made by management 
regarding the following areas . 
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• (1) 

(2) 
(3) 
(4) 

(5) 




Managing an Evaluability Assessment 



This manual describes the series of-activities which we believe to be necessary in completing an 
evaluability assessment. Two things must be said about managing such an enterprise. First, the 
sequence of such activities described here should not be taken as the only proper one. While it is 
logical and orderly, it is sometimes made impossible by the reality of events. Much of the information 
needed must be obtained from policymakers, managers, and program operators, all of whom are 
busy people, and it is usually up to the evaluator to change his schedule to suit theirs. Second, many 
steps or analyses defined separately here are often done together, and as often rather more 
informally (some might say haphazardly) than is implied here. Even using our method of analyzing 
programs prior to evaluation, no two completed assessments look alike. 

We expect that many evaluators who attempt to follow this manual will discover that our rules and 
procedures are imperfect, and we that new or different procedures will be developed by those 
who attempt to use the method. Still, it must be remembered that often the evaluation team will have a 
fixed budget and schedule to complete the EA. Within these constraints, the project will have to adjust 
resources and products to produce an assessment that serves management best, a judgment that 
the evaluator should make jointly with program management. Immediately following are some guides 
to Planning and controlling the work, and subsequently, the reader will find some specific how-to 
guidance 

Planning an Evaluability Assessment 

The evaluation team will spend most of its time m the following activities: 

• Managing the EA: planning the work, scheduling activities, controlling 
and adiustmg staff iillocation to activities, monitoring and controlling 
the quality of the work; 

» 

• Reviewing the literature: reading and becoming familiar with the 
legislation, heanng records, program plans. gr Jelines and regulations, 
research and evaluation reports, documentation of field activity, etc.; 

♦ Interviewing policymakers (includes upper level bureaucrats; Office of 
Management and Budget staffers. Congressional aides of Senate and House 
authorization and appropnations committees), program management, ano a 
sample of regional office staff and State stqff (when appropriate); 

♦ Documenting program descriptions obtained from literature search and 
interviews and communicating with key officials to verify the accuracy 
ot the descnptions. 

♦ Conducting site visits to observe program activity for the purpose of 
describing direct and indirect intervention: 

• Completing analyses to determine evaluability; 
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• Documenting (in briefings or written form) the conclusions and 
supporting evidence; and 

Organizing and administering the decision process following from the 
assessment. 

Dunng this period, management is expected to participate also. Their activity will include: 

• Providing program description to evaluator; 

• Assessing the validity of the evaluator s descriptions and conclusions; 

• Providing guidance to the evaluator on the relevance and direction of 
the work; and 

• Making decisions on options develo Ded by the evaluator to resolve 
problems affecting evaluability. 

Table 1 indicates roughly the level of effort t..at might be expected for these activites. T!ie budget 
for the evaluation team in this case is estimated at 8 person-months; the time available. 4 calendar 
months Dunng this time, the evaluation team must slay in contact with management, providing 
information as the effort proceeds. 

Controlling the Work 

Controlling the work is perhaps no different from controlling any other analytic effort, all of which 
have a way of expa.iding as the work progresses. The team leader will have to judge whether a 
particular activity is beginning either to exceed the expected value or diverge from the main path. A 
;,'mbiance of control is provided by tracking the consumption of man-hours on each activity and 
tr.K king schedules As these are flexible, however, a team leader cannot expect too rigid an 
•jf ihcroncc to original estimates— except that maintenance of overall cost and schedule constraints is 
srtHK^rtant 

\\v assume that more than one person will be involved in any EA. and have, in fact, found the 
:>firtujm toam size to be two or three people. One obvious advantage of such a team is that if one or 
•'. r,. members become stu'^k on some issue or problem, the group can jointly decide what to do 
li'i 'ut It Often we have found ourselves stuck on some point and discovered either we needed more 
.:u H ific (l.ita or were trying to make some decision that was not ours to make. As a group, we were 
it!!»- to (loterrnine that contact with program management was necessary. 

f if. jiiy I ontrol is afforded by frequent progress checks against a checklist of activities and 
products Such reliance on the list is important, simply because it is so easy to drift away on 
intercstinq questions or a quest for more conclusive data The evaluator also must always bear in 
'inrnJ that the effort is nof a complete evaluation study— rather it is an evaluability assessment. He 
•nust do only enough to allow himself to move to the next stage. While the potential data and analytic 
• iui'stions are limitless, the time and resources, as well as the patience of program management are 

ni it 



TABLE 1 TiiiM Spmit on EA AoU^ititi in 
• Four C«l«nd«r-Month P^iod 



Acti¥iti#t 

Managine the EA 
Reviewing Literature 
Interviewing policymakers 
Documenting Program Descriptions 
Visiting Sites 

Completing Analyses 

t 

Documenting Findings 

Organising the Decision Process 
TOTAL 

Activities 

Providmg Information 

Judging Validtty 

Provtdmg Guidance 

Makmg Decisions 
Total 



Evaluators' Time 

(Person-Weeks) 

2 
4 
. 4 
4 
4 
6 
4 
4 

32 

Management's Time 
(Person-Weeks/Person) 

2/5 

1/10 

1/10 

2/5 
1 



II. THE FIRST STEP: 
DESCRIBE THE PROGRAM 



Evaluability assessment is a 4-step process: ( 1 ) describe the program; (2) validate and analyze the 
program description to determine the extent to which the program can be evaluated; (3) construct 
alternatives wherever necessary; and (4) implement management decisioris. 

This first step ts crucial in evaluation planning and program management. There is often a 
substantial discrepancy between the concept of a program ias perceived by managers and 
policymakers and the reality of that program as it exists in the field, as well as a hesitancy on the part 
of management to admit to the discrepancy. Part o i t he p rqblem is no doubt a simple unwillingness to 
admit that management at the Federal level does not andl>robably cannot control what happens at 
the loc^ level in most programs. Acceptance of this truth may seem to threaten their positions and 
seriously weaken their support in Congress. / 

Yet continued unwillingness to accept the differences between the reality and the idealized 
perception^ of those programs can seriously limit the ability of management to obtain useful 
performance information and influence or control the events and outcomes. It is for this reason that 
we have developed an approach to evaluation planning that provides a method to describe programs 
and determine the extent to which such discrepancies exist. We do this within the context of program 
management in as nonthreatening a manner as is possible. The method is intended to help program 
managers gain whatever control is possible— to help them achieve their ends« as well as satisfy the 
demands of congressional and executive branch oversight functions. 



What Is a Program Description for Purposes of 




The core of an evaluability assessment is program managements description of the program. This 
involves two elements— real programs and various descriptions of real programs. Descriptions are 
almost invariably highly condensed summaries of real programs, which nearly never capture all 
activities and effects involved in a program. This lack is exaggerated by the fact that program 
descnptions are often inaccurate, reflecting what people think, want to believe, or want'others to 
believe is actually happening. To the extent they do not incorporate key program characteristics 
necessary for evaluation or are inaccurate, we believe that evaluations based on those descriptions 
will be equally inaccurate. Thus, in evaluability assessment, we continually check various program 
descriptions against one another and against the main referent— the program as it is observed to 
actually operate 

What Is Bemg Described in an EA? 

For purposes of an EA. the program description should cover those activities which management 
can guide and control to achieve national objectives. Those activities can be broken out as: 
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Direct /nferven/»on— activities designed to directly change or serve 
society in some desired way: 



Program Accountability—actmiies through which management 
acts to report performance and establish realistic expectations among 
policymakers. 



Indirect Intervention —acimies through which manao^ment acts to 
develop and alter the direct intervention and its perf^mance; 




The components of each are illustrated in Figure 1 . 



The direct intervention is usually at the interface between government and the recipient of the 
government service Examples include community mental health centers and their clients; Head 
Start projects and their pupils; Occup'?.tional Safety and Health Administration regulations, in- 
spectors, and the workplace; and the L.S. employment service offices, the job seeker, and the 
• employer. 

Indirect interventions are a type of overhead function whose purpose is to assure and improve the 
performance of the direct interventions. Evaluation systems, technical assistance programs, regional 
office monitors. State and local planning agencies, research anb demonstration programs are all 
examples of indirect interventions. The term " indirect" is used because the indirect intervention must 
first change the behavior of some other organizational unit before program performance can be 
altered. 

Program accountability can occur through formal decision processes such as budget and over- 
sight procedures or through informal mechanisms oUch as publications, meetings, correspondence, 
and conferences. Federal program offices establish staff activities to deal with both the formal and 
informal channels of communication with the mission of enabling management to get agreement with 
policymakers on realistic expectations ior the program and informing policymakers on progress and 
program performance. 

What Comprises a Program Description? 

Programs can be descnbed in many ways, but for purposes of an EA certain specific types of 
infomiation should be included. Our scheme for describing programs consists of elements that 
constitute a description and properties or criteria that each must satisfy to be judged evaluable. If one 
or more elements are missing or one or more criteria are not met. the probability increases that 
evaluation will be useless and measured program performance unacceptable. 

Elements The following elements are required for a descnption (see Table 2) 

• a sequence of events: 

• an event description: 

• measures and comparisons defining each event activity. 

• expected values for each event; 

• activities that must exist for event to occur; 

• information system used to provide evidence the event occurs; and 

• intended use of evaluation information. 

Event sequences are used to define the ordered chronological interrelationships among events. In 
part the sequence defines the logic— the assumed cause and effect relationships among events. It 
also implies an order in which measurements could be taken. Finally, sequences suggest possible 
stopping points in measurement; if we know that Event A has not occurred, we may not need to 
measure Event B. 
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PROGRAM SYSTEM SUBJECTED 
TO EVALUABILITY ASSESSMENT 



COMPONENTS OF AN 
EVALUABILITY ASSESSMENT 



PROGRAM MANAGEMENT 
DECISIOfJMAKINQ FUNCTIONS 
BEING SUPPORTED 



Polcymakers 
(Program Oversight) 



Management 



Indirect Intervention 



> 



Direct Intervention 



Chenges in Society 
Attribuied to Program 



An EA to Build 
Management Capability 
To Justify the Program 
and Evolve Realistic 
Policy Expectations 



^n EA to Build 
"^Managemen^ CapabUHV 
To Influence and Guide 
Program Operations^ 
Specifically the Direct 
Intervention and Its 
Effects 



An EA to Build 
A Valid and Useful 
Measurement Framework on 
the Direct Intervention 
for Accountability and 
Management 



Accountability 



Program Planning* 
Guidance, and Control 



Performance Evaluation 
(Effectiveness and Impact) 



Changes Not Attribut- 
able to Program 



FIGURE 1 



Distribution of th^ Tasks in an Evaluability Assessment 
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'owner" tefe^s to person or group from whom mforntatton is obtained, e.g., program manager. 
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Event descriptions are used to define the event, as in ^children from X*type families will read as 
well as children from Y-type families/* Attempts should be made to avoid event descriptions which 
include or imply two events, as in ' childrea from X-type faniilies will read as well as children from 
Y-type families as a result of adopting Z-type curriculum.** Such events should be described in at least 
two statements. 

Measures and connparisons are used to define the type of evidence that will signal event 
occurrence In the event above, the measure might be a comparison of performance on a standard- 
ized reading test in children from X-type families with children from Y-type families. Comparisons can 
be one group against another, one group against itself over time, or both, 

4 

Expected values are used to define the direction and amount of change expected, if we say, 
reading scores will increase **thenimplicitlyanyinc*'easeisacceptable. A better statement would be 
that reading scores will increase 25 percent in one year or that scores between two currently unequal 
groups will become equivalent. Often it is useful to define any expected change in a comparison 
group to avoid a situation in which, for example, reading scores become equal because of a reduction 
m comparison group scores. 

Activities are used to define and help in examining the plausibility of the basic program design. 
Subsequontly. measures of the types and amount of activity can be used to help explain perform- 
ance. 

information systems are used to define the sources which will be used to collect data, especially as 
to whether the sources now exist or new systems will need to be created. 

Intended use helps to define and examine the probability that evaluation information will be worth 
Its cost. 

Element Properties. The event itself must begin with an activity over which management has 
control, conclude with activities* representing the primary intent or objectives of the program, and 
include all of the key events necessary for the objectives to be achieved. Selection of key events is 
judgmental, so evaluators and others must be selective in defining the necessary, and sufficient 
conditions for the objectives to be achieved. 

In addition to the properties. EA requires an analysis of the program description to assess the 
extent to which the description satisfies evaluability criteria based on eliminating typical problems 
which have prevented or inhibited useful evaluation in the past. The following standards must be met: 

• Well defined— the description ts well defined when it satisfies the 
requirements of T able 2: 

• >^ccepfa5/e— the description is acceptable when it matches policymaker 

expectations: 

• Valid ~\he descnption is valid when it accurately represents the 
program activities actually operated in the field; 4 

• Plausible --the description is plausible when there is evidence of 
plausibility and no evidence of implausibility; 

• Data feasibility— the data system defined in the description is feasible 
when It does not impose cost or political burdens t)eyond those which 
management poltcymakers are prepared to accept; 

• Data ri)liability—ihe data system defined in the description is reliable 
when provisions are built in for repeat observations and further 
verification jjrior to use; 

• Plausible use of /nformaf/on ~management*s expectations for the program 
and the evaluation are plausible when its use of the information is 
defined in terms of known resources and activities within its control. 
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r«iI5!J!if procedures In Chapter III. Table 3 illustrates the 

relationship between tftdse EA criteria and the elements of a description. 

Models: A Useful Language Convention In 
Describing and Analyzing Programs 

We use three types of nuxlels to describe programs; 
• Logic models: diagrams representing the intended logic of the program, i.e., 



Evt»nt 




Event 


A 




B 



If Event A occurs, then it is assumed Event B will occur, with events 
representing program effects or objectives. 

• Function models: diagrams representing the intended activities or 
functions of the program which support, produce, or lead to the intended 
events, i.e.. 



Activity 
1 



Activity 
2 



Event 
A 



• Measurement models: diagrams representing the points at which measures 
can be taken, as well as the types of measures needed to represent 
activities and events. I.e.. 



for 
Activity 
1 



Muasur es 

for 
Activity 
2 



Measures 
for 
Event 
A 



TA^LE 3 Evaluabtlity Asuumtnt Crittria 



tv«»!ii Stt<|ut*nce 



EvtMit Descitptton 



Criteria 



# Acceptable 

# Well Defined 



• Acceptable 

• Well Defined 



Acceptable 
Wei! Defined 



Acceptable 
Well Defined 
Plausible 



* • Acceptable 

EvkI.„cv • Defined 



Cost-Feasible 
Known to be Reliable 



Acceptable 



l)v Marwn.ru J Well Defined 

• Plausible 



Each of ihe models can be used tpBStablish the extent to which the program description satisfies the 
EA criteria (Tal)Je.4). 

TABLE 4 Um of Models in EA 



EA Criteria 


Logic Model 


Function Model 
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Measurement Model 


Vtfell Defined 
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Acceptable 
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Valid . 
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Plausible 
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Feasible Data 






X 


Reliable Data 






X 


Plausible Use 
t>f liitoffnation 




X 


X 



It should, of course, be remembered that other types of descriptions are possible, although we 
advocate develoRment and use of these models in an EA. The key issue is not the method used, but 
the information it elicits, as well as the adoption and use of the EA criteria. 

Logic Models 

We noted that logic models are of the type: 



B 




A Simple example would be: 



If 


t hildi ar e 






vai:c:ir>ated 





then 





children do 




n<)l*contract ' 




smallpox 
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or 



If 



^ 30 percttPt of 




children are 




vaccinated 





then 





smallpox will 




be eliminated 




in the region 




Figure 2 illustrates a logic model (Schmidt et al.. 1 976) developed to describe the health and child 
development program operated by the Appalachian Regional Commission. The model represents 
the intended events connected with one part of the overall prograiTi— health and child develop- 
ment—and was used to define this specific program priortothedecision regarding an approach to 
evaluation. ' 

Function Models » 

Function models are intended to 'define the program in temns of sf}ecific activities and flows of 
people, mon#y. and information. Often called process or stock and flow models. <hey are intended to 
capture the operations or processes which characterize a program or project. 

Figure 3 illustrates a function model ^developed to define one part of a demonstration program 
(Horst. 1978) funded by the Law Enforcement Assistance Administration (LEAA) — Managing Crimi- 
nal Investigations (MCI). The function represented is case processing, the flow is information. 

Measurement Models 

Logic and function models can be used by themselves for defining, analyzing, and comprehending 
a program of some complexity, but measurement models can only be used in conjunction with the 
others To illustrate the intenelationships of the models, we have excerpted material from a report 
completed for LEAA (Nay et al.. 1 977-Annex D) concerning the use of measurement models. Rgure 
4 illustrates a function model for a city methadone treatment programJjy showing people flowing 
through various functional activities (e.g.. in supen^ised methadone treatment, event 6). 

In the accompanying table, each state or condition of the fjbpulatlon involved is defined, the 
measures are identified, and current availability of a data collection system is noted. This model is a 
relatively simplified representation of such a program. 

« 

Figure 5 then represents the (ogic and measurement models 'for this program. The events and 
assumptions at the top represent the program logic. The evaluation questions and measurements 
and comparison rows represent questions of Interest and measurements to be taken from the 
function modei. Note that the numbers in the measurements and comparisons entries represent 
measurement points taken from the function model. Figure 4. and that some of the questions ask for V 
descriptions of how whether certain events occor. Note. too. that existing data systems are in- J 
adequate to answer all questions. 

In conclusion, it must be again stressed that the models are nothing more than a language used to 
define* certain aspects of a program. We find models useful for our analyses, but other forms tiave 
been and are used to define programs. 
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FIGURE S Logic and M^MuriiMnt Mod^l for M«thadoM TrtatiiMiit Prooram (Typa A) 
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What Are the Sources of Information for a 
Program Description? 



Since an EA assesses the program management's description of the program, its completion 
requires management s description plus the descriptions of policymakers, evaluators, and program 
operators for companson. Each source may consist of several people, so there may be more than 
one program descnption from it. 

A program manager at the Federal level is someone with responsibility for those activities 
established to achieve some national objectives. Whether or not the events comprise a program is 
arbitrary and a matter of policy. A law, for example, could be brol<en up into five programs with five 
managers and five objectives, or set up as one program with one manager and five objectives. One 
may find a single program with one objective spread out among several departments with several 
managers who act as one to insure that the national objective is achieved. This diffusion of 
management responsibility makes EA all the more necessary to bring the pieces together. 

Program management then, is made up of the manager(s) of a program and those people he or 
she includes m the management decision process. Typically the participants will be an agency 
director or bureau chief plus the supervisors of line and staff units reporting to them. The manage* 
ment group usually is three to nine people who meet and communicate frequently. 

Policymakers are those people with some oversight responsibility for the program, Including the 
power to qrant the manager legal authority to administer the program. Even for a major Federal 
program the policy group usually will not exceed 12 people. Its members are: 

• Program Assistant Secretary. 

• Under Secretary and or Secretary.. 

• Office of Management and Budget examiner. 

• Government Accounting Office audit division director. 

• Staff directors of House and Senate budget committee and oversight 
committee 



For any given program this group may vary. The important point is that in a few weeks one can usually 
t.jik to overy government official with a significant policy voice in the program. 

Tht> policymakers define value for the program— thai is. those program outcomes judged to have 
s(H t<^l tH>nofit Comparison of theiulescriptions with those of program management is the basis for 
thr acceptability criterion ^ 

Program operators are those people running specific pieces of the indirect and direct interven- 
tior^s Thoir description allows another companson to estimate validity and plausibility, r^rogram 
iHH?rator d^xscriptions of the program can be oieced together from interviews with the operators 
fosponsible tor the different activities 

Tho evaluators are the people doing the data collection, analysis, and feedback for the EA— 
usually a team of two to four people. Their description is based on direct observation. When com- 
pared with program management s description, it allows assessment on the validity, plausibility, 
ancl cost feasibility criteria. 
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Table 5 defines the probable sources of information for each type of model; Table 6 defines the 
information items carried in each. 



Getting Program Management's Description 

There are eight steps to complete to obtain a description of the program from the management 
group Following is an explanation of each. 

Identify Key Actors; Bound the Program 

There should be only one program manager and one program involved in an EA. The tasks of 
defining the key actors and what will be considered as the program are interrelated. 

Analysts have had substantial trouble defining the programs of government for two principal 
reasons First, the laws enacted by Congress and the subsequent budgetary appropriations are often 
broad collections of intents, activities, and resources, sometimes covering whole agencies, so that 
there may be many programs within any single law. Second, the concept of a program is fundamen- 
tally a question of definition and perspective as seen by policymakers. The concept implies a set of 
activities being managed by a single program manager to achieve a common end and, therefore, it 
has utility only for t;he purpose of fixing responsibility for achievement. 

There are some general critena which could be applied to the determination: 

• The objectives are national, and somehow reflect congressional intent; 

• There is a single manager responsible for achievement of the national 
intent. 

• The manager has not delegated all of that responsibility to someone 

else, and 

• There is a defined set of resoiJrces and activities presumed as adequate 
to accomplish the end purposes, and the manager has Access to them. 

Gonerdlly there is a level within a department below which national objectives lose their identity 
and above which the concept of measurable, achievable ends is lost. That level is where programs 
cue found For example, within DHEW the bureau or institute level is where programs are most likely 
to bo found Above that level, there are agencies or management levels whose leaders are con- 
cornod with poUcy issues and the management of program managers. Below that level, the effort of 
more than one unit (a division) would be required to achieve national objectives. 

Tru> key actors in program management include the individual designated as program manager 
ami his irnmodiato subordinates, e g . a bureau chief and his division directors. In some situations 
prinjfcUTi management might include rec^onal office staff who are directly responsible for administra- 
turn i)f some part of the national program 

In the bvsi possible situation the program manager has line authority over the resources needed to 
ai hifwo the ncVtional objectives In government, however, this authority is often divided. A classic 
t^xarnpie is the Occupational Safety and Health Administration (OSHA) and its research arm. the 
National Institute of Occupational Safely and Health (NIOSH). OSHA managers, presumably re- 
siH>nsiblo for administration of the Occupational Safety and Health Act. are part of the Department of 
Lcibor whiio NIOSH personnel report to DHEW A more common example is the situation in which 
(irant administration a major part of a program has been decentralized to the regional offices, which 
rtHM>rt adrrtinistfatively to someone other than the program manage In this case, the program 
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TABLE 5 Typical Sources of Infornuition for E«ch Program Modal Type 



Dascriptton 


Type of Information in the Detcriptioh 


Logic 
Models 


Function 
Models 


Measurement 
Models 


Policymakers 


Legislation 
Hearing: 

Committee reports 
Interviews 


Legislation 
Hearings 

Committee reports 


Interviews 


Management 


Regulations 
Guidelines . 
Interviews 


Regulations 
Guidelines 
Work plans 


Guidelines 
Data systenr. 
manuals 

Evaluation designs 
Interviews 


Operator 


Interviews 


Interviews 
Work plans 


Interviews 
Files 


Evaluator 


Observation of 
program activity 
and flows 


Observation of 
ptogtam activity 
and flows 


Observation of 
program activity 
and flows 
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manager must form agreejpents with other offices to insure that he gets the needed resources. If he 
does not have line authority over his resources and cannot negotiate an adequate priority to obtain 
them from someone else, the program manager must present his problem to upper level managers 
and policymakers for resolution^ 

The Identification of key actors and definition of the program will bound the work of the EA and 
make clear by definition whose views will represent program management. Note that we generally 
exclude from that group other management levels and organizations (the offices of the Assistant 
Secretary and Secretary. 0MB. Congress). Their personnel will be included in the group defined here 
as policymakers. Their descriptions are also required, but in a separate step. 

Collect Descriptive Materiajj^^ 

Individuals within the mangCgem^X group must^eHthe eVakjator whatspecific material should be 
reviewed Generally, the j?§cef^ry material is included in the followinq: 

« 

• legislation 

• legislative hearing records (these are often helpful in defining intent) 

• program plans 

• program budgets 

• regulations 

• guidelines 

• research reports. 

It IS not unusual to review 30 or 40 individual documents that, in the aggregate, describe the 
program Often State and local plans are helpful in putting together a coherent description. 

Prepare Rough Models 

The condensation and translation of a large body of descriptive material into model format requires 
a substantial amount of judgment on the part of the evalua lor— basically, selecting and sorting for key 
events and activities Of necessity, some events will bo excluded. There will be seve-al opportunities 
to verify your selection and arrangement with •nanagers. policymakers, and the like. 

Often it IS helpful to begin by prepanng a program structure, a chart resembling an organizatbn 
chart but which lists key objectives, subobjectives. and activities needed to achieve the objectives 
•nstead of organizational units. Initially, it is easier to prep.-^re a logic model and use it to begin 
preparation of the function and measurement models. These models need not be presented to 
anyone at this stage as they are intended merely to assist the evaluaior in constructing his interview 
guides 

Identify Key Events and Gaps 

In final preparation for the interviews with program management, the rough models are reviewed 
-.. ,mci <in attempt is made to identify areas and events which seem to be Incompletely described. The 
focus of the interview should be on these areas. 

lntervieM£ Key Actors 

It will be helpful to prepare a guide before each interview, spelling out the information you want to 
get from the interview However, be prepared to listen carefully to everything that is said and take 
(Jetailed notes In fact, we have found it helpful to have two people conduct each interview, so that at 
least one compl'.te set of notes will be produced. 
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After the interview, the rK>tes shouM be written up as soon as possible. Of course it is important to 
try to avoid injecting your view into the notes. Reinember that real people with* responsibility for the 
program are your source of descriptions. To insure that the description is accurate, send a copy to the 
person you interviewed, asking him to confine it. 

The completed interviews should provide enough information to define logic, function, and meas- 
urement models. If this is not so. you may have to schedule reinterviews to obtain specific pieces of 
information. 

Prepare Final Models **"' 

From all the information at your disposal, prepare logic, function, and measurement models. 
Remember, though, the models should be treated as a language whose purpose is to communicate. 
The three models are used because they describe different aspects of the program. What is 
important is their infonnation content, not their form. 

Analyze the Model: How Well Defined Is It? 

The critena and procedure to employ in judging the extent to which the program description is well 
defined are described in Chapter III. 

Verify With Management 

Your description (the models) and assessment should now be communicated to management. We 
recommend a briefing rather than a written report to save time and increase communication. 
Management should either agree with the findings or sugges* specific changes to the description. 



Getting Policymakers' Descriptions 

Getting a program description from policymakers follows roughly the same series of steps as ms 
outlined for the program management group and is done primarily to determine the differences 
between the two views. Policymakers, however, are generally a larger and organizationally less 
related group and include the following members: 

• Agency head 

• Assistant Secretary 

• Secretary 

• 0MB budget examiner 

• Congressional committee members 

• GAO representative. 

Whenever possible, the actual policymaker should be included but be prepared to settle for 
proxies, such as an aide to the Secretary or congressional committee staffers (at least one staff 
member from minority and majority sides for both authorization and appropriations committees from 
both houses). 

The following six steps comprise the activity'. 

Identify the Key Actors 

Get a list from the program manager of the key officials who should be contacted. 
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Analyze Descriptive Material: Prepare Rough Models 

The rraterial coflected earlier for the program management description should contain all of the 
needed documents for the policymaker description. Key documents include legislation ana hearing 
records. Office of the Secretary directives. issue analyses, top level plans, and budget. 

Again the purpose of the models is to prepare for the interviews. The models may be rough, 
although often the legislation itself can provide substantial descriptive material of a functional nature. 
Figure 6 was prepared early in an EA as a functional modeL based heavily on examination of the 
legislation, and is in more detail than most such models derived at this point. It had not yet been 
validated, but it proved helpful in many subsequent discussions and interviews. 

Interview Key Actors 

Interviewing policymakers often presents special problems although this is less true for outside 
consultants tnan for government personnel. We have always found such interviews to be valuable. 
However, govemjrent personnel have indicated that while consultants may be able td, hold such 
interviews, the/do^ot have the same access to the players. Furthermore, the congressional aides 
'WtUaJ:«v«jIjck6ated that they welcome such opportunities are viewed as relative rarities. 

Prepare Final Models 

Based on the review of written documentation and the interview notes, the evaluator should 
prepare final logic, function, and measurement models to the extent possible. 

Compare Policymaker and Program Management Models 

The criteria and procedures for analyzing the extent to which policymaker and program manage- 
ment descriptions of the program agree are outlinecl in Chapter III. 

Validate Differences 

Whenever differences between the two descriptions seem apparent, they need to be validated. 
The first point of contact should be with the program manager, who is asked whether he agrees or 
(iisaqroes with the different descriptions in the policymaker models. Should he disagree, it will be 
re(vss<uy to consult with one or more of the policymakers to be certain that your understanding of 
their intent is valid Any policy disagreements discovered by this process will have to be resolved by 
the program manager with his superiors. 



Getting Program Operator's Description 

Thv proqram operator is typically a grant recipient who provides some service to the public under 
< i^nditions laid down m the law, regulations, and his individual grant. The program operator might be a 
lorai leqa* <ud office director, a health systems agency director, or the director of a State planning 
aqen( \ In the health planning program (Public Law 93-641). each of the individuals administering 
State health planning and development agencies and the health systems agencies (HSA s) is a 
pro()r^<^^ operator 

Tvpicaiiy their description of the program will reflect a relatively narrow focus, the perspective of 
kHWiv Yet if the program is to work nationally, the efforts of local program operators must add to the 
n«Uion<ii objective There is only one effective way to get their description. Go onsite. 
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Seletting Sites ^ j. « 

' The selection of number and location of sites is important because what 13 ^ught Is a picture of the 
program as it really operates. Of necessity, however, one needs to compromise because of the 
constraints of time and budget. In the health planning program, five sites werQ.visite(l for^ days each. 
In similar work for the Appalachian Regional Commission (ARC), eight projects, six ioc^l planning 
units, and two State program offices were visited to obtain a description of the program' from the 
perspective of local program operators. 

\ . ^ ' • • 

Th^ number required depends paftially on how much variation is expected in fundamentally 
differeht type€ of activity. For example, in the health planning program, only two types of field 
organizations, were involved— one at the State and one at the local level. For ARci there were State 
units, local planning units.. and 30 to 40 different typfes of sen/ice providers.' • 

Preparing For thc^ Site Visit 

Preparation for the site visit is similar that for interviews with program rtlanagers and policymak- 
ers In this c^se llbwever. the documentation has been predigested. Principally, there will be a grant 
application, local plans, and the set of models derived earliej^ which provide cludsto help in focusing 
the onsito* work. The purpose is to determine the oxtent to which the national program is supported by 
local programs— in other words, do the two seti of descrtptions match? 

Review of the available documentation, including models, should t>e used to determine what to 
look for onsite Basically, you will be looking for': 

• objective statements. 

• measurement indicators. 

• intervention activities. ^ ' ""^ 

• operating assumptions. S \ 

• project resources. * ' . . 

• external intervening variables, anc^ 

• data sources. \ 

♦ • 

\ 

In preparation for the site visit, you should draw Upan agenda outlining the objectives of your visit and 
send it to the people with whom you will meet. ^ 

Conducting the Site Visit ' . ^ ( ' . 

Site visits will generally involve two evaluation personnel and last 2 or 3 days (up to a maximum ^f 5 - > \ 
days ♦or a complicated program). While onsite. the evaluators §hould brief the director and hisstaiff on 
tho Dbiectives of the visit, which will be pursued by interviewing the director and his staff, observing 
<inci r€?cordinq actual processes, rr^'owing data, plans, records, and local mformation systems, and 
prt.'parinq flow charts of the operc 1 

Bofort? I oncludinq the sito visit, it 1.". wise to brief the director on your findings in an attempt to verify 
vi>iif (1»»S(.fiption of his piogram Try o obtain the following types of information. 




Obiective Statement— OMa\n narrative statements of the objectives of 
thp proqram as defined by field manaaement personnel. In obtaining 
thf?se statements, try to get definition of both the immediate output 
( biectives expected as the direct result of intervention activities and 
th«« realistic expectations of intent. 
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• Measurement Indicators— Qe\ the management staff to detail their best 
judgment on what measurement indicators they would accept as plausible 
success measures of their objectives. The defining of measurement 
indicators is an iterative analytic exercise involving the evaluator and 
local program management for the purpose of reaching some tentative 

. agreement on the meaning of the objectives in measurable terms. 

• Inten/ention Activities—\n obtaining a description of the actual 
activities being conducted by local personnel, it is important to define 
the actual intervention activities being conducted onsite, as well as 
the relationship of activities to each other cind to the prc^ram 
objectives It is expected that local programs may differ substantially 
in terms of the activities conducted and the manner in which the 
activities are conducted in relation to other ongoing activities. 

• Operating Assumptions— \Nh\\e onsite. attempt to obtain the views of 
management on the chain of assumptions that led management to believe 

V that tile intervention activities would lead to achievement of 

objectives You will need to add your own assessment and views of these 
assumptions later, but the management of the program can substantially 
enhance .your understanding of the idiosyncratic conditions that may have 
led to the adoption of one approach ovei- others that are followed in 
different sites. * 

• Program Resources— Obtain as accurate an estimate as possible of the 
resources actually deployed by the program. The accuracy of such 
information will undoubtedly vary among sites, depending upon the cost 
records maintained and the extent to which" the local program is operated 
independently of other nonproqram activities. Document the basis used, 
including records consulted, r determining the allocation of resources 

to activities 

• External Intervening Variables— MempXio identify any variables 
outside the control of the program which could plausibly affect the 
validity Its operating assumptions 

• Data Sources— Attempt to define and review the data systems maintained 
by the local program, obtaining copies of records and reports. The 

local programs will often prepare internal reports which do not get 
forwarded to Washington. 

• ' Protect Flow C/)ar?— Develop rough flow charts that describe the program 

as a linked set of activities and objectives from the expenditure of 
funds to the achievement of intended impact. Be careful to avoid 
tntfoducing or modifying such factors as objectives and causal links, in 
an attempt to introduce greater logic or tidy up the program design. 
This task is not intended to describe what should or could exist; 
rather it must capture what does exist. If there are significant 
differences between what you have been told and what you observe, try to 
verify your observations while onsite. 

Developing a Generalized Program Operator's Description 

Using the seJ of local descriptions developed onsite. devise a general framework for describing the 
program This task re(^uires synthesis of the matenal collected from individual sites into a program 



model representative of what was observed in the field. This model is to represent as closely as 
possible what actually exists, and must emanate entirely from the descriptive material you obtained. 
It IS entirely conceivable that analysis of the individual programs will reveal such disparity among 
programs that no overall model can capture the observed programs without serious distortion. Some 
possible outcomes of this task include these observations: 

• Local Programs Are Idiosyncratic— Tfye field programs surveyed reveal 
such dissimilanty that an overall model is not feasible; 

• Sets of Programs Emerge— There exist several models of sufficient 
simthinty that generalized models can be developed; or 

• One Model Predommates —There exists such ovenwhelming similarity among 
focal app'-oaches (even if not all programs follow this approach) that 

one generalized deschption appears useful. 

Compare Program Managemenrs and Program Operator's Descriptions 

Cntena and a procedure for examining the extent to which program management's description is 
supported by. or conflicts with, that obtained from the program operator is provided in Chapter III. 

Communicate Findings to Program Management 

Using a briefing format, communicate the results of your analysis to program management as soon 
as possible 



The Evaluator's Description 

Generally the evaluator will not prepare a separate set of models or descriptions, but rather will 
make changes to the other models which he is shown If an evaluator description is prepared 
formally it is a composite descnption. gleaned from the others he has reviewed. 
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III. THE SECOND STEP: 

ANALYZE PROGRAM DESCRIPTION 



Is Program Management's Description 
Weii Defined? 



Rules To Follow 

The basic rule is that adescription of a program is well defined when jtcontalnsall of the elements 
stated in Table 2. Chapter II. Table 2. as already noted, is a listing of the Information needed for a 
complete program description and criteria to help the evaluator judge the adequacy of the Informa- 
tion The table can be used in the form of a simple checklist (Se^ Table 7). whereby the evaluator can 
judge, event by event, whether the descriptive material he has at his disposal is adequate; his 
conclusions can be recorded on the checklist with a simple yes/no. 

Based on our experience, the elements in Table 2 constitute the minimum Information set needed 
by an evaluator to complete an EA. Other evaluators may feel the need to add elements which they 
behftve to be necessary. Indeed, in our experience we have found the need for additional informatton. 
The desire is to manage a program to achieve specific results. 

Procedures To Follow 

• Begin the examination of the program description with a checklist, the 
criteria outlined in Table 2. and the program descnption. 

• Using the model formats, extract needed information elements (using 
Table 2) from the program descnption. 

• Make judgments on whether or not the program description includes all 
needed elements. Record conclusions on the checklist. 

• Review conclusions with owner" of program description (e.g.. program 
manager). If new information is offered, alter conclusions. 

• Document final agreement t>etween yourself (as evaluator) and owner on 
final conclusions regarding how well defined the description is. 

Using the Models. We have already stressed the value of using models as a kind of shorthand 
description of what we know about a program. We assume here that the evaluator has already 
coiiocted as much basic descriptive information as he can. and that he has constructed the three 
types of descriptive models. 

Using the models and a checklist derived from the table of information elements, the evaluator can 
go through the checklist for each event in the logic model, making yes/no decisions on each element. 



33 



TABLE 7 Checklist for Racording if a Program DMcription is Wall Defined 



INFORMATION ELEMENTS 


Event 1 


Event 2 


Event 3 


Event 4 


1. Event sequence 


2. Event descriptions 


Yes/Nc 








3. Measures and comparisons 


Yes/No 








4. Expected values for each event 


Yes/No 








5* Activities that n^.u*>t occur for 
event to take place 










6. Information system used to 
provide evtdet^ce 










7. Us*?s of evaluation itifurmatiof^ 
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For any event tn the logic model, it may be necessary to refer to one of the other two models to 
determine whether the logic model event is well defined. For example, an event in a logic model might 
be the cost of hospital care in Region X is reduced/* The table of information items asks whether 
there is a measure and a comparison defining this event. By referring to the measurement model, the 
evaluator might find a measure, ' average per diem cost acceptable by Blue Cross in 1979 compared 
With the same cost in 1978. * For that event in the logic model, then, the evaluator could indicate 
yes. ' the event has a measure and a comparison base. (Note: Subsequent analyses might reveal 
the measure to be unacceptable for one reason or another. Our concern here, remember, is only with 
a determination of whether or not the description is 'well defined/' that is. it contains all of the 
necessary information elements.) 

Similarly, it will bo necessary to refer tp the functional model to determine whether each event is 
defined m terms of trie activities that must occur for the event to take place. Table 8 summarizes the 
source by type of model for each of the information elements in Table 2. 

Expect to Find Gaps. Going through the checklist for the logic model as a whole and for each event 
m the model separately provides the evaluator with a picture of the completeness of the description. 
For any program, it would be extremely unusual to conclude that all information is available especially 
on the first pass at a management description. The expected condilion is that as you move from item 
1 (event sequence) to item 7 (uses of evaluation information), more partial information and complete 
gaps will appear For a new program, for example, there may be no information system and no well 
defined uses for information. Measures and comparisons also may be only partially thought out. with 
expected values defined by 'increase,'* "decrease, * etc., rather than specific quantitative values. 
The functional logic may be only partially thought out. (What events affect per diem 
hospital costs and how are they calculated?) 

The Significance of Gaps. Several points should be made here. First, identified gaps at this stage 
may proye to be unimportant on further analysis. As an example, consider a simple case of a 3-event 

model 



Event 1 

Op»*».t'»- *l»iv raif 
t •*tit»»r tor 



Lvent 2 

M<>thi»is III poot 
Uimilit»s Will be 
iibit» to woiR 



Event 3 

RiHjion X Will 

t»t:onomic impact 



Via the checklist, we Conclude that events 1 and 2 are well defined, but event 3 is not because there is 
no measuro or measurement system Suppose, though, that we check with the policymakers and 
they declare that they have no concern about event 3. If we can demonstrate success at the event 2 
lovet. that is adequate. The info mation gap disappears, and the event should be eliminated from the 

scheme 

Gftps can also be filled in by subsequent work. For example, a gap in defined measures for some 
t*vont in the management descnption might be filled in after a visit to program operators in the field 
fovoals that they know how to (and do) measure the event. The point is that before you plug every 
qap check first to make certain that (a) it is a gap, and (b) it is important. 

Socond there are two apparently different types of gaps: (1 ) those in the logic or function of the 
program design itself; and (2) those in the information part of the program (the absence of a 
measurement system) The evaluator might consider the second type his province: the first type 
someone else s On closer examination, however, the distinction gets fuzzy. The evaluation design is 
an integral part of the program design, and the evaluator becomes part of the program design 
process whilo he is completing an E A In some areas, to be sure, the evaluator lacks the technological 
pxpertiso to fill gaps, but in others, he will be the expert. In either case, he should be prepared to 
participate m the process of defining alternative program design options when necessary. 
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TABLE 8 El«m«nti of a Program Description that can bt 
Extracted and Oiipiayad by Oiflartnt Modali 



Information Element 


Logic 


MMturtment 


Function 


tvt»nt st*c|ut*ncH 


X 








X 


X 


X 




X 


X 


X 


EH^)f»i:tfnl valutas 


X 


* 




Actfv t.fN thai must oa ui 






X 


Infdiniation svstt>ms 


X 


X 




lnt»'rul*»cl LueH of ifittvniation 


X 
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Example 



The following is an example of how we examined the extent to which the manager's description 
was well defined in one program. One part of the Bureau's health planning program is its technical 
assistance strategy, illustrated in Figure 7 as a logic model. The boxes represent events that were 
expected to occur, the arrows indicate causation. The strategy assumed that the Bureau would 
develop products (e.g.. planning methods) which the health system agencies would use to achieve 
changes in the health care system. 

Figure 7 has been simplified for purposes of this example. It represents the actual logic model 
. constructed to represent the Bureau's expectations, asconfirmed by management. The assumptions 
represented by the model would be tested by an evaluator. Do the events occur? Are expecta- 
tions me\^ 

It ts important to note that this model is not a set of research hypotheses dreamed up by an 
evaluator but rather a policy statement— the logic used to justify the Bureau*s technical assistance 
program to Congress. 0MB. and DHEW. 

Regarding the question of how well defined is this logic model, a simple yes/no checklist presents 
the conclusions reached after examining the program description (Table 9). The "yes" boxes signify 
that the event description satisfied the required information elements; the "no** boxes reveal partial or 
complete gaps 

Table 10 summarizes the information available on the technical assistance (TA) program. The 
analysis revealed that program design had never included the last two events, except to nanrte them. 
Missing from the design were processes whereby agency use could be defined and therefore be 
known Also missing was a process which could be used to track changes in the agency caused by 
the program products and link those changes to health care system changes. 

This program was judged to be unevaluable on the last two key events. Note that this finding would 
not prevent the program from being implemented, but rather that it indicated there was no way to 
know whether or not the TA was useful. Instead of simply communicating this finding to manage- 
ment, the evaluators worked with Bureau staff to define an alternative design which satisfied the 
critona of a well defined program (see Chapter IV). 

Note that the gaps in the lar>t two events were made more real by the agreement between 
policymakers and program managers on the need to measure success at the level of events 3 and 4 
tn Figure 7 Had policymakers and program management agreed that success was defined by event 
2 the cjaps would have disappeared 

Is Management's Description Acceptable to 
Policymakers? 

Th^> purpose of this assessment is to determine whether there is agreement between policymakers 
< ind program managemerit on the substance of the program One should not expect the same level of 
detail in both, but rather that they agree on the main intent and strategy being followed. 

Rule To Follow 

Thf » rule to follow is that if events in the policymakers* program descripuon also appear in program 
m«in«iqemen* s description, you can assume they agree. 
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BUREAU STAFF ACTIVITIES 

(About 20 percent of Bureau 
staff resources assigned to 
its technical assistance program) 

1. 



PRODUCTS ARE 
PRODUCED AND 
DISSEMINATED 
TO AGENCIES 



PRODUCTS ARE 
USED BY LOCAL 
AGENCIES 



3. 



USE OF PRODUCTS 
RESULTS IN 
REDUCTION OF 
OVERSUPPLY OF 
SPECIFIC HEALTH 
SERVICES 

4. 



FIGURE 7 Expectations for Bureau's Technical Assistance Program 
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TABLE 9 lllustr«ti«« CiMoklUt: How Wail OtfiiMd it Program OMcription? 
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Expected value for each 
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Yes 
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1 Activities that must occur 
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No 
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Description of operation 
used to take measures 


Yes 


Yes 


No 


No 


Who will ijse evidence 
of event occurrence? 


Yes 


Yes 


No 
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Information system used 
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Yes 


Yes 


No 
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Uses of evaluation 
information 


Yes 
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4. Expected value for each 
event 
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resources and 
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The two descriptions will have been obtained independently, and therefore, differences between 
the two may or may not reflect disagreement. If the difference is that events appearing on the 
policymakers description do not appear on program management's, further interviews should be 
held with program management to determine if the events were merely overlooked. On the other 
hand, if events on the program management s description do not appear on that of the pc 'cymakers, 
the decision on whether to reinterview will depend on what is reflected by the omissions— differences 
in level of detail or in apparent intent or strategy. A simple checklist can be used for this task. 

Procedure 

The basic procedure to follow is to (a) compare the logic models of the policymakers and program 
management, (b) conduct any reinterviews that seem necessary to validate differences, and (c) hold 
infcimal briefings for the purpose of highlighting differences and detennining whether they reflect 
disagreement m intent or strategy. 

The evaluator is here trying to obtain agreement on a common description so that he can proceed 
with the EA. Disagreements may require the collection of further information until agreement on one 
model IS possible. 

Example 

Figure 8 is an example of a program description (presented as a logic model) based on interviews 
with a policymaking group. Comparison of this logic model with management's description led us to 
conclude that there was agreement. ' 

Those above the Bureau agree with the Bureau in their 'oectations foV 
Program IIJ^ Using the results of our interviews with Dep<tiiment of Health, 
Education, and Welfare (DHEW). Office of Management and Budget (0MB). and 
congressional staffs, we were able to construct Figure 8. a model of the 
pr "gram from the perspective of those above the Bflreau. This model is 
Similar to the Bureau s view of Program III. 

Both the Bureau and those above the Bureau expect Program III to affect 
Significantly the operations and capabilities of health systems agencies 
(HSA s) and State agencies. 

Figure 9 illustrates the logic of this program as now reflected in program management and 
policymakers descnptions. and our own assessment of how well defined was the logic. Note the 
confirmed gaps in the logic— Events 3 and 4 have disappeared. Had policymakers and program 
managers not both agreed on the need for these latter events as evidence of success^the gaps would 
have been unimportant. As it was. they had to be filled. 

It would be helpful at this point to be able to cite some instances of disagreement between 
policymakers and program management. Unfortunately, m the few cases in which EA has been 
applied fully, few disagreements of this type have occurred. One program in which a difference 
emerged was in the Appalachian Regional Commission s (ARC) health and child development 

program 

A substantial part of the rhetoric defining the Commission's program, especially in the area of 
hoalth and child care, suggests that the Commission is intended to fund new ideas, not merely to 
supplement DHEW funding. Interviews with congressional aides revealed that congressmen took 
this View of the Commission s program seriously, and did not believe that the Commission had made 
a serious attem;;i to do anything very experimental. 
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FIGURE 8 Developing and Dissemmating DiscreHonary Technical 
Assistance Products, Logic Model of Policymakers' Description* 
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FIGURE 9 Comparison of Program Management and Policymaker Logic with 
Assessment of How Well Daf iited was the Program Description 
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Figure 10 (Schmidt, et al. 1976) illustrates the logic models defining the innovation objective 
Analysis of the model indicated that the objective itself had never been defined in measurable terms, 
and that the Commission s statistical reporting system by design did not segregate innovative from 
conventional projects The Commission had never developed any processes for implementing the 



The situation as revealed by the EA. was that Congress had expectations not shared by the 
program management group within the Commission. Note that the EA did not conclude that the 
Commission failed to sponsor any innovative ideas: in fact, it did. The conclusion was only that there 
was no formal process whereby innovations were separately identified, no overt Incentives to 
encourage the development of innovations, and no information processes to study and disseminate 
information about innovations. 

A tentative alternative was developed to add an information process. The alternative was subse- 
quently adopted and subjected to further analysis to define the objective and an operational ap- 
proach, thus bringing the Commission s program more nearly into line with the policymakers' 
(Congress) views 

-\ 

Is Management's Description a Sound 
Representation of the Program In the Field? 

The spatial and organizational distance that separates Washington from real program activities in 
the field creates a special problem for managers and evaluators. It would not be unusual to find 
arguments raging m Washington about objectives, strategies, and the fine points of evaluation 
research that Simply disappear when program operators meet and service the public. Before 
attempting to answer the question. Does it work?" it is wise to ask. "What is it?" Does the program 
debated m Washington exist in that form in the field? 



Rules To Follow 



Table 2 identified the sets of information that constitute a program description for management and 
evaluation purposes Two sets describe the program activities which must exist onsite to produce 
results measures and compansons defining each event; and activities that must exist for events to 
take place 



This part of the E A estimates whether or not management s description of these events accurately 
rt>pr*'sents the program as it operates. The rule to follow in making this determination is tf.it 
managf >m<>nt s description of the program is valid when the field activities observed and described by 
tho f'vaiuator conform with management s descnption of them, and management's measures and 
comparisons are sufficient to descnbe accurately the activities there. 

fhf. ri,i»! ,s applied by companng management s descnption with one or more independent 
(lf s( riptioMs of \he activities Since the operator is closest to the program and the evaluators 
desc ription r^jprosonts an impartial. Oirect observation, these two descriptions are likely to be correct. 
When disaqrf!(>ments arise, additional onsite observation can determine which of the three de- 
scriptions IS valid 

rat)ic> 1 1 1S a checklist to record the conclusions reached in applying the rule. The evaluator should 
oe prepa^fKl to support eac h conclusion by presenting material from the «ippropriate program 
(1»'S( ripticn 
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FIGURE 10 Program Modal ^or Damomtratina Innovation 
an Appalachian Regional Commission (ARC) Project 
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Procedure To Follow 



• Take management s description of program activity and the measures and 
compar4sons associated with each event identified. (Measurement and function 
models can be used to facilitate extraction of these elements from the 
description.) 

• Compare management s description of activity with what operators say exists 
onsite and with what the evaluator has observed onsite. 

• Make a judgment as to whether managements description is a valid one. 
Record the conclusion in Table 11. 

• Compare management's description of measures and comparisons required to 
descnbe an event with what the operator and evaluator believe is required to 
capture the behavior of actual activity onsite. 

• Make a judgment as to whether management s measures and comparisons are 
sufficient. Record the conclusions in Table 1 1 . 

• Review conclusions with program management and note areas where management 
agrees, disagrees, or requires further evidence. Continue providing 
evidence and discussions until there is agreement between management and 
evaluators on conclusions. 

• Document final agreement. 

The program operator s (or evaluator s) function model represents the activities observed by the 
evaluator to exist onsite which should be compatible with that of management. Differences should be 
verified by the evaluator to insfure that the fie'd models are correct. ' 

The program operator s (or evaluator s) measurement model represents those sets of measures 
which the evaluator believes would represent the activities and outcomes of the observed program 
and those sufficient to define the events of interest to management. Again the operator's measure- 
ment model shouJd be compatible with that of management, and verification may be necessary 
should important differences be observed. Conclusions regarding measurement potential (those 
measures common to both) and measurement problems (differences observed) can then be re- 
ported 

This particular companson can be used to avoid an error frequently made by evaluators— 
measuring something which does not e>:ist It is an area in which cleverness and a compulsion to 
produce elegant analyses can lead evaluators to define their own measures and measurement 
systems which may not represent the real activities and intent of a program. 

Example 

Ev<iiuatK>n studies completed for the Atlanta public school system provide an example of differ- 
tnves between a program operation as perceived by management and as it really operated. In this 
(Mse neither spatial nor orgarnzational separation was as severe as is encountered in most Federal 
proiir<irrts yet differences in perception were still significant. Note that the Atlanta example is an 
•r^staru e m which EA criteria were not used until near the end of the project, and then only to find out 
wr^v the system did not work as intended Consequently, we failed. 
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Figure 1 1 illustrates a simplified logic model of the school system. It shows the school administra- 
tion making decisions regarding subjects such as assignments of teachers and other resource 
personnel, training, and various sanctions and rewards. These processes are expected to affect 
teacher and overall school performance, reflected by student performance measured through 
standardized tests 

A relatively simple signaling system was devised, whereby test results from classrooms were 
plotted and converted statistically into above and below average performance for students from 
relatively comparable econonnic backgrounds. These results were translated into signals— symbols 
representing student performance in various schools and classrpoms. It was assumed that such 
information would be used to target the various resources. 

The evaiuators did not develop function models of the actual processes and decisions used to 
make the assignments, and it was discovered aftenward that performance had not been and still was 
not used as the basis for making these decisions. Table 1 2 illustrates a checklist that might have been 
prepared for Atlanta had" we conducted an EA. Note the areas in which management's description 
proved to be incorrect or incomplete. Had we known of that problem, we would have been able to stop 
at that point until we reached agreement with management on the need to introduce new activities 
and measures 

To use such a performance signaling system, the school administrators would have to develop and 
adopt a different method of operation. First, the signals would have to be used as triggers releasing 
personnel skilled in classroom performance assessment. Next, these personnel wouW have to 
compare leaching styles and skills to determine what factors seemed to contribute to above or below 
average performance in specific classrooms. Next, correctives would have to be developed and 
applied Finally, performance changes would have to be measured. These functions did not exist, 
and consequently, the evaluation system failed because the evaiuators did not verify the validity of 
management s description of the system actually used to correct performance problems. 

It stiould. however, be noted that a different interpretation is possible here. Everyone knew that 
resource teachers were not assigned according to performance criteria, but it was assumed that the 
criteria would change with availability of the signaling system. The evaluation demonstrated that the 
criteria did not chang< . and therefore, it was successful in pointing out this fact. The authors believe 
this outcome could have been predicted and avoided by designing new activities and criteria. 



Are Management's Expectations Plausible? 



However well defined is the description of the program, judgments about plausibility remain. Many 
have .uqued that plausibility judgments should not and cannot be made until after a carefully 
designed evaluation study has been completed, but we feel they should be made as early as possible 
and prett>rably m advance of an evaluation study. We maintain that arguments against such analyses 
arv akin to lakinq a stand against the need or possibility of completing technical feasibility studies in 
advance of buildinq a bridge 

Rules To Follow 



Twt) of the elements in Table 2 describe how the program is expected to perform: event sequence 
.ind expe( tations The plausibility analysis in E A asks what is the likelihood these expectations will de 
re,iii/ed with the program in place or being set up"? If the likelihood is high, the expectations are 
labeled plausible, if the likelihood is low. they are implausible. 

The rule to tollow m making this determination is that management s expectations are plausible 
when there ts some evidence that the program activity onsite will actueve ihe rebuUb expected, and 
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FIGURE 1 1 Logic Model for the Administration of the Atlanta School System 
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Invaitisation 

Would 
Unoovar and 
Fix Things 
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School 
Pefffonnanc# 
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and 
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t Program management has 
Tdentified activities that 
must exist onsit^ 


Yes 


Yes 


Yes 


Yes 


2. Program nnanagement has 
identified measures and 
comparisons 


Yes 


Yes 


Yes 




Yes 


a 
1 

1 


3. Management's activity 
descriptions are valid 
representations of program 
activity onsite 


Yes 


Yes 


No 


No 


4. Management meaiiures and 
comparisons are valid 
representations of activity 
onsite 


Yes 


Yes 


No 


No 



there is no evidence to the contrary The types of evidepce that can be used are research or 
evaluations showing what results these activites produced in the past, the opinions of experts and 
operators, and the judgment of whether the program activity is large enough to create the expected 
result 

Table 13 is a checklist to use in applying the rule. The basic rule requires observation and 
judgment The key is to look for evidence thai the program is plausible, with the absence of 
convincing evidence making an argument for implausibility. Remember that "implausible" is not a 
judgment that something will not work; it is merely an argument that there exists either no or 
inadequate evidence that it will. 

Procedure To Follow 

• Management s event descnptions and expected values for events are 
associated with performance (output products, expected results). 

• Compare management s expected values with those obtained from the 
operator or evaluator descnption. 

• Make a judgment as to the relative plausibility of management's 
• description 



Record the conclusions reached on Table 13. Record evidence used to 
"^uppf^rt conclusion. 

• Review conclusions with program management and note areas in which 
disagreement exists. If needed, obtain additional evidence from the - 
field and continue to work with management until agreement is reached on 
a plausible description. 

A>;st iming the two sets of event deschption* correspond, the basic test is performed by comparing 
' f'xp(H;ted values for each of the events. Note that the program operator's description to which we 
refer should have been converted to an aggregated description, especially regarding expected 
values The issue is that assuming all programs operated in the field perform at their expected values, 
will ifioir aggregated performance meet or exceed the expected values found in the management 
description'^ 

X;{ inclusions arc obviously easier to reach in cases where necessary activities are not in place at 
all Aith()U(}h such situations have m tact been observed, the more likely situation is one in which 
suftici»^'ncy ts the issue or in which the technology is auestionable. 

Whorever threats to achievement appear, the evaluator should reconfer with program operators to 
l)t; certain that the apparent threat is real. The evaluator should note for each event whether it is 
pi.jusiWe or implausible, and what is the evidence tor making the determination. 

ObvKJusly analysis of plausibility is likely to be different for new and ongoing programs. The most 
otjvious differonce lies in the existence and relative experience of program operators. Brand new 
pf<Kjrams will have no current program operators, so a vital comparison is missing. In such cases, the 
ovaiuatcr will have to rely on review of the literature supporting the program, on extrapolation of- 
management s expectt.'d values from single sites (in a program where many sites will contribute), and 
nn the .uiaiysis of resources likely to be available tor the program While the available information will 
ih> U'ss and the conclusions necessarily more speculative, the analysis is no less useful in such 
situations 

Example 

Tfic findings that might emerge from an analysis of program design plausibility relate to the general 
no'iufi (;i ilu' ii«<>;ssary and sufficient set of conditions that must exist for a program to succeed. 
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Necessity argues that some definable set of conditions, both internal and external to the program, 
must exist. Sufficiency argues that these conditions must be prf'sent in the right quantity. Conditions 
are resources (money, people, facilities, equipment), activities, and other factors affecting perform- 
ance. 

One example of an EA leading to doubts about plausibility concerns the Bureau's technical 
assistance program—a subprogram of the overall health planning program (Public Law 93-641 ). The 
conclusion emerging from the EA was that this program was unlikely to result in the (xxxin'ence of 
certain events (specifically, events 3. 4. 5. 6 in Figure 8). Here the analysis produced certain 
conclusions about the extent to which the necessary set of conditions existed. The conclusions 
rested on several observations: 

• Interviews with Bureau staff revealed that they were unsatisfied with 
their own rate of producing information and the information products 
themselves; 

• Interviews with HSA staff—the main recipients of the Bureau's 
products— revealed that the products were not highly thought of; and 

• Examination of the process being used by the Bureau revealed no 
activities designed to promote use or to assess the relative 
effectiveness of the TA effort. 

Each of the findings was documented in detail and presented to management, which was asked to 
decide whether they agreed with the findings, and to determine whether the available evidence was 
sufficient to warrant action. Further, some alternatives were offered (see discussion in Chapter IV) to 
stimulate their thinking. 

A second example is available within the same program. As a result of the EA. Bureau manage- 
ment agreed that an interim objective for the program was HSA development of evaluable programs 
to produce national guidelines. It was assumed that development and implementation of evaluable 
programs was one of the necessary conditions to achieve success, but examination of the plans 
developed revealed some disturbing findings. 

For one. HSA s were adopting objectives, and presumably expending some of their resources, in 
hundreds of v^reas. Two. although the plausibility of individual program plans could not be assessed 
conclusively, the information in the HSA documents strongly suggested tfilSt many of the objectives 
were not supported by real activity. Finally, the tentative conclusions about plausibility conformed 
with earlier interviews with HSA directors who stated that they knew many of their objectives were 
implausible, but that they were in the plans to conform with DHEW planning guidance. 

On the basis of very limited evidence, we projected that the Bureau would in several years be 
unsatisfied with the program s achievement. We suggested that there would be considerable 
anecdotal evidence of small successes at the local level, but that there would be limited evidence of 
success in areas of national interest. We speculated.that the lack of limited and clearly defined priority 
areas and planning guidance which stressed the need to complete "comprehensive " planning had 
led HSA s to a position in which their efforts were spread thinly over too many areas. 

Wo suggested the need to focus HSA s more narrowly on areas of national interest, as well as to 
focus the Bureau s information collection process to access information in stages, with each stage 
being more narrowly tuned on selected areas than the preceding one. The first stage, for example, 
might use a telephone survey of a sample of HSA s to determine areas in which HSA program activity 
promised to be successful. Successive stages might then involve collecting more detailed informa- 
tion about the potentially successful local efforts. Even the initial stage, however, could be made 
more specific by selecting prionty areas of national importance as defined by Bureau policymakers. 
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The strategy was intended to produce information about those parts of the program which seemed 
on the basis of objective evidence to be plausible, thus reinforcing those parts. The important point 
here, though, is that even limited evidence of plausibility can cause management to redirect its 
activities. In this case, many people within Bureau management were already predisposed to such a 
finding. In other cases, more conclusive evidence might have been demanded. The key is to feed 
back such evidence as is available, noting its limitations. As an evaluator. you cannot assume that 
management will demand conclusive evidence before acting. 

Is Management's Evidence Reliable? 

• In preceding steps, we have detemiined^that management's measures are valid, that is. they 
measure the right variables. We must now be concerned with the extent to which the evidence to be 
used IS reliable. If it is. the instrument and measurement process will produce the conect values for 
any subsequent management actions to produce the desired result. Repeated measurements by 
independent observers which produce the same findings increase management's confidence in the 
reliability of the measurement system. 

Rules To Follow 

Management s evidence is likely to be reliable when the following conditions exist: 

• There is an independent measurement taken for comparison, prior to 
action based on information; 

• The information is used in such a way that several observers can comment 
on reliability, and 

• There is a capability to get additional information to resolve 
re\ 'iity issues. 

Procedure To Follow 

The basic procedure to follow is: 

f 

• Examine management's description of the measurement processes to be used 
for each maior event. 

• Ascertain whether there exists a process for obtaining repeat 
observations of events prior to any actions or decisions by management. 

• Interview management to determine how many repeated observations are 
necessary to trigger management action. 

«. 

• Record ludgment as to acceptability of measurement process relative to 
management s intended use of data. 



Communicate ludgment to management. Propose alternative reliability 
measurement processes if required 



Example 



Legal Services Project Evaluation. The Office of Economic Opportunity (OEO) Office of Legal 
Services implemented a project evaluation system involving review of the quality of legal work 
performed by its grantees. The procedure involved use of an instrumer»t for recording observations, a 
team of trained legal observers familiar with the program, and a sample of case files from attorneys in 
the grantee s office. 

Cases were selected randomly from the attorney s open and closed case files. Using ain interview 
ctjeeklist. attorney-raters Interviewed the attorney about each selected case, and numerical quality 
. ratings were assigned Two raters were used to produce a consensus rating for each review. 



Then, before any management action was taken, successive, independent assessments were 
Conducted by regional office staff and a second site visit team. Further, reactions to the findings were 
obtained fronn the grantee. 

*• Based on »>ie findings, one of two management actions was to be taken in each case. For high 
quality grantees, grants were to be expanded to provide more service, and grantee staff were to be 
used as sources of technical assistance. For low quality grantees, grants were to be terminated. In 
such cases, assessments were repeated over a period of more than 1 year, and on/y cases in which 
grantees showed no. improvement were grants canceled or not renewed. 

In this example, the instruments— rating sheets and interview guides— were incidental, their 
reliability unknown. The key 1o the evaluation was attorney-rater, judgment, and the process of 
consensus-seekmg through repeated, independent reviews was crucial. Also, an important factor 
was that the evaluation process only had to be sufficiently accurate to detect very good and very bad 
quality. Had it been used to rank order performance of all grant&es. Its reliability might have been 
questioned. But since most grantees performed in the midrange. the risk of making a wrong decision 
here was quite low. given repeated observations and adequate time to correct deficiencies. 

Solid Waste Coliection. Another measurement system involved assessment of the overall effec- 
tiveness of municipal solid waste collection programs. As one part of a larger assessment process, a 
p' jcess and instruments were developed to rate the cleanliness of city streets a.nd alleys. The 
applied measure was the percent of streets and alleys rated satisfactory on a visuai ••Ptinq scale 
/Hatry. 1977). 

Here ratings are based on visual ratings by a specially trained observer. The cleanliness and 
appearance of a street or alley are graded in accordance with a set of photographs and written 
descnptions that cover a range of litter conditions generally found throughout the community. Trained 
observers driving along the streets (or through the alleys) assign to each block face a numerical rating 
that corresponds nnost closeiy to a grade described in the photographs and written descriptions. The 
observer does not have to leave the car to make the ratings. 

These ratings constitute a readily unders' . neasure. especially when results are presented with 
photographs on which the rating scale is based. Under proper supervision, the trained obsen/er 
program provides a reliable way to measure aesthetic Impacts on the community and changes in 
( itMniiness over time Photographic rating scales for cleanliness have been used by the govern- 
ments of Washington. DC. New York City. Savannah. Nashville, and St. Petersburg. (Further 
information on the New York and Savannah efforts is available in reports from both these cities. ) The 
information thus gathered can be tabulated to indicate cleanliness by nelghtiorhood. sanitation 
st>rvice area or for the entire city The visual rating procedure, despite its attractive features, can 
(iiiiokly degenerate into an unreliable procedure unless proper care — particularly in supervising 
inspt»ctors IS taken. Over time, inspectors may forget the definitions and become carele<js in their 
<ipplK:ali(.)n iir even in making ratings In one city, the ratings of one inspector gradually retreated into 
so narrow a spectrum that he did not use the extreme ratings regardless of the circumstances. (This 
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scaLe compression is relatively compfion when inspectors dc not adhere to photographic or written 
standards.) A second city was uftable to use its ratings after it discbvered that independent but 
virtually simuitaneous inspections of the same scenes had resulted in wide ratinf disparities that had 
not been promptly corrected by retraining th^ inspectors. f 

Experiences to date indicate that several actions are needed, particularly after implementation of 
the rating system . to insure that the data are reasonably reliable. First, the development k the visual, 
photographic rating scale should be undertaken systematically to assure that the rating scale can 
reliably distinguish the different levels of cleanliness. Second, adequate training of observers in the 
use of the rating procedure, both for new observers and as periodic retraining for experienced 
observers, should be provided to prevent deterioration In rating skill. 

Finally, regular checking of sample ratings should be done to determine if obsen/ers maintain 
sufficient accuracy In their ratings. Perhaps 10 to 15 percent of each inspector s ratings would be 
replicated by the supervisor. The local government should set reliability targets; for example, 76 
percent of the inspector s ratings could be required to agree with those of the supervisor's, and 90 
percent to agree within one-half point. If the checks do not indicate sufficient accuracy, immediate 
retraining of raters and possibly further refinement of photos and written standards for the ratings 
should be undertaken. 



Is Management's Evidence Feasible Td Develop? 

The question of information fea,sibility is heavily, although not exclusively, a cost issue. Some- 
times, other issues, such as confidentiality, can ovenwhelm t)asic cost. 

Rule To Follow 

Management s evidence is feasible to Oevelop when the cost of the information system falls within 
reasonable operating and evaluation budgets and the information is politically acceptable to develop. 

Procedure To Follow 

Tables 14 and 15 are illustrative checklists to use in examining the financial feasibility of infomia- 
tion systems 

The basic procedure is to; 

• Estimate the costs of collecting, storing, processing, and verifying the 
information for each event. 

• Calculate rough system costs m terms of staff (internal) and dollars 
(contract cost) 

• Estimate the cost burden on program operator staff (if they are required 
to supply data) Venfy with operators 

• Rt^viow system data collection process with those expected to supply data 
iqonorally program operators). Determine relative acceptability of data 
collection scheme 

V 

• Communicate findings to management to determine whether total burden 
unirludinq operator costs) is acceptable 
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TABLE 14 Simple Chteklitt for Dttermining Coiti 
A w oda f d mih Running • Data Systatn 



Cost of D«ta Syitem 

Contract Staff 

A. 



Data Collection 




• * 


Storage 






Processing 

(bv ) 
• 

• • • 


% 

• 


• 

m 


Clu Jl It V Check 

■ ^ 

1 


1 





57 



TASLE 15 Sample Chtcklitt for Dtterminina Cost FMSibiltty 



Event or 
Description 


Availability 
of Required 
Measurement 
System (Yes/No) 


Measurement System Cost 
$ staff time or unknown 


Mest u r A mefit 
Method for 
Checking Errors 
(Yet/No) • 




Total Budget 
for Event 


A 






• 


• 


B. 










C. 










D 










E 










Total Cost 











The cost of information systems is almost inevitably a subject of controversy in government. The 
0MB forms clearance function was developed expressly to provide a check on infomiation prolifera- 
tion Often, the only cost included in analyses of systems is that incurred in developing a system 
under a contract, while costs of implementing the system, checking its quality, or paying program 
operators are generally not included Frequently, operator systems, already in place, are ignored, 
thus increasing the expected cost of the system because of required modifications to operator 
systems The price of fgnoring such costs is that management can be led to make decisions they will 
later have to reverse 

Example 

An example m which all factors were not accounted for is provided by the Legal Services 
Corporation (LSC) lSC and its progenitor. OEO s Office of Legal Services, had been criticized/or 
some time by the General Accounting Office for the failure to implement a basic information systam, 
LSC was in the process of mounting a series of demonstrations to assess alternative legal services 
delivery methods and one part of the effort involved the collection of best and workload data. Under 
contract a project reporting system was developed to collect the information required by the 
ovciluation dosiqn The system involved adoption of uniform case file forms in each office, the use of 
t»niqut. client idefitifymg numbers, and the recording of time invested by individual attorneys. 

The system relied on the collection of disaggregated case-related data being forwarded to 
hp ac1()ij:irte'^s for processing and analysis. It imposed costs on individual attorneys, projects, and the 
corporation and added costs to monitor reliability. Having developed the system, however, LSC was 
tht>n uHcible (o implement it. Important among the many is iues raised was the subject of potential or 
tvarpd vjoiatton of client confidentiality pledges. Further, project directors believed the system 
intruded tco far .nto their domain, threatening their autonomy. ' 

F vvn if the actual financial cost had proven acceptable to management, a possibility because of an 
')ffor ot fin ncial assistance made to LSC. the fX)litical costs proved to be unacceptable. A com- 
ptomise system had to be and was developed. 

It should bv noted here that there exist few usable ground rules for determining reasonable costs. 
Thv so called 1 -percent policy, m whicn the Secretary of DHEW is authorized by Congress to spend 
up U) 1 percent of program budgets for evaluation, is not a guide. There are instances in which over 50 
percent of a demonstration program budget was spent for evaluation, and others in which 1 percent 
w(niid tH* high In fact, there is no substitute for costing out the whole information system, even quite 
roiKjhly. and allowing management to determine whether the cost is too high. 

Are Management's Expectations Plausible Given 
Their Intended Use of Information? 

Manaqemi^nt s expectations derive, m part, from the basic design and assumptions of the program 
and \t\ pvut trorn their presumed abtlity to redirect the program based on evidence of performance. 

Rules To Follow 

Management s ex^)ectatlons are plausible when there is evidence that deviations from expected 
jH^rtof mance c<m be identified by management and that management can affect program activities to 
n^rroi t U.r performarn^e deviations Table 16 provides a checklist to use in reaching conclusions 
UHHit manaqemont s use of information as a fe€>dback device to correct performance deviations. 

Procedure To Follow 

Ih*» prtM:t?duro to follow m this caso rt.»quiros the evaluator to interview management about planned 

uso 
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TABLE 16 ChMklitt to AiMtt Planned Um of Information 



^ Affenment 


Event 1 


Event 2 


Event 3 


Event 4 


1. There is evidence that management 
has defined acceptable performance 
leveis t « es/i>io;. 


• 


• 






X There is evidence that manage- 
ment has onsite activities (or 
plans) to correct performance 
when it deviates from 
acceptable performance levels. 










3. Management planned use is 
plausible (Yes/No). 












ERIC 



# Interview management to obtain a definition of the perfomiance levels 
that would tngger either direct action or further inquiry, as based on 

a description of expected values for key events, and a definition of the 
existing or planned activities controlled by management which are 
believed capable of affecting the performance of the event(s) in 
question. 

# Analyze the functional description of the program and attempt to link 
management's planned actions to functional elements which govern the 
event performance. 

# Note any gaps between management's planned actions and the event in 
question. 

# Form judgment as to the plausibility of managements intended use of 
information, and record them on a checklist as per Table 16. 

# Communicate findings to management. Continue to work with management 
until a plausible management event sequence is deveioped to affect 
performance. 

In practice, the evaluator is being asked to extract from management another program 
description— thisoneconcernedwiththeindirectinten/ention.Theprogrameventsmust,again,l)egln 
with resources and staff activities over which management has control and end with the event(s) 
whose performance is to be corrected. 

This last analysis is. if not ignored, at best completed halfheartedly in most evaluations. The 
problem seems to stem from at least two causes. First, managers often do not know exactly how they 
intend to use information and, thus, cannot tell evaluators. Second, there continues to t>e a rather 
generalized belief that performance infonnation will automatically tell management how to correct 
problems. 

This latter point is especially problematic. It may follow from the general notion that evaluation is a 
method of assessing experiments and that findings are supposed to be unequivocal. Yet few Federal 
programs are experiments in any way other than name alone, and most evaluation contracts ask the 
contractor to ^toduce findings and recommend solutions to observed problems. It Is our general 
contention, to the contrary, that solutions must come from those responsible for managing programs. 
Further, it ts our belief that at least the outlines for plausible solutions should be developed before 
rather than after an evaluation study is completed. 

Example 

We cited earlier the example of a performance signaling system developed for the Atlanta public 
school system which failed because the evaluators did not define the real functions carried out by 
school administrators and resource specialists. In other words, had we known the real program 
activities, we would have said that the program expectations were implausible. 

This same study serves as an example of the issue being discussed here. The planned manage- 
ment use of the information could not have affected performance as intended because management 
had not designed its indirect intervention to use such information. Instead, use was left relatively 
undefined It was assumed that performance information would trigger allocation of resource 
teachers, but it did not It probably was assumed, too. that technology existed (teaching methods 
known to be effective) which would correct performance deficiencies, but it did not. What was 
required here was. as already noted, a fundamentally different type of indirect intervention — one 
qeared to research and development— in which performance deviations were only the first step in a 



61 



multistep. corrective process involving the study of successful methods and the development of 
technology and methods to use it. 

The earlier cited Office of Legal Services quality assessment method, on the other hand, was 
plausible, precisely because management had carefully designed its planned use of information in 
such a way that the actions were plausible. OEO. in fact, used the system as planned because its 
planned use already matched existing functions. The information system merely formalized a 
process already m place 

The point is planned use cannot be treated lightly, a fact that is amply demonstrated by even casual 
reading of most evaluation plans prepared by agencies. We close this subject with material obtained 
from a prior study of evaluation planning (Hoist et al.. 1974a). 

The FY 1974 tDHEW Evaluation Planning) guidance asked the program offices to specify the 
issues which evaluation should address. This was done to help clarify exactly how evaluation was to 
be used However issues in most submissions were portrayed in such general terms that it was hard 
to tio them to any particular decision context Typical issue statements included: "Determination of 
overall effectiveness of various types of^««ivices in relation to cost under various conditions." 
••devolopnient of sy stenn management approa^es to improve sen/ices," and "the relative effective- 
ness of current and proposed mechanisms for dWivery of services in the public and private sectors." 

The guicMnce also asked respondents to specify users and to associate users and issues in order 
of pn(>fity We found that most program audiences defined almost everyone in the world as potential 
users of their evaluation studies For example, one program office specified the Congress, the 
Secretary of DHEW Health Services and Mental Health Administration, the National Institute of 
Mental Health the program director, regional offices. States, local project directors, and other 
qovornment and private agencies. No priority among them was indicated. To illustrate how another 
program office described evaluation users, their written responses arc presented in schematic form 
in Fiqurt* 12 The boxes represent users at the Federal. State, and .local levels. The arrows show 
decision pror esses or products that evaluation was supposed to affect and the groups that would be 
aftect<Hi 

Th<r wst* of irnlivKJiiai evaluation project findings was treated in general terms, as the following 
fjiKitatu^rw tfom plan suumissions illustrate 

• Knowlpdtje assist NIMH. States, and facilities to plan 
us»*ful data svstf»rns 

• U\ hOK al assibtanct* tind f)laruiin(j 

• x]u iUfM p \(^ State and tt)unty (jovernments <ind increased NIMH 
.iruj M*»<]K)nal OttK.e eVtiUiativr (:apat>ility 

• f^t»sults •)! !h»s stuflv will rissist planners and project directors to 
ff<liM»( t th*-<r prcHjfcirn » for bett^^r ai:hiev« 'tnen* of their goals and 

«»l)|tM tivt*s 

br used »>v.«lu.itK >ns rnusf hi* treat(?d as new f)roducts for whose us<? substantial design 
t»ntjint'f»r'fnj vvofk nujti'fed It cannot bv left to chance 
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FIGURE 12 Schematic of Usen and Uses of Evaluation Based on the Division 
of Narcotic Addiction and Drug Abuse. FY 1974 Evaluation Plan Submission. 
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IV. THE THIRD STEP: 
CONSTRUCT ALTERNATIVES 



Early application* of EA led to an unsatisfying conclusion— the one or t\vo potentially evaluable 
descriptions concerned studies of relatively low interest to management, while more interesting 
studies were blocked by the EA conclusions which suggested the need for further definition or actual 
program design changes. It seems, then, that EA could prove to be a highly effective brake on 
expenditures for evaluation. This particular outcome, while somewhat appealing, does not take into 
account that the evaluator, having created an apparent impasse, should be the one to define 
alternatives which are helpful to management. It is probably also true that an evaluator's conclusions 
regarding problems in program descriptiorts coukl be strengthened by defining and offering man- 
agement ways to resolve the obsen/ed problems. 

We define, then, an added task for the evaluator doing an EA— construct alternatives wherever 
possible These can take the form of alternative expectations, program activites, or information 
approaches For any of the threats to program achievement named throughout, evaluators can 
suggest one or more of these alternatives to fill gaps in the program logic. Often, the alternative might 
be a developmental process of some kind to discover and define an appropriate alternative. 



Alternative Expectations 

Perhaps the most common problem affecting Federal programs is unrealistic expectations, most 
likely resulting from a management decision which is invalid, implausible, or unacceptable to 
policymakers. 

For any one of these problems, one possible solution is altered expectations. Most Federal 
programs are defined legislatively with a specific budget level and a relatively limited definition of the 
specific activitiesr that will or must be present to fulfill the intent of the legislative and executive 
branches Often, the EA will be the first time anyone seriously examines the underiying logic of the 
design and the implied logic of the real program as it is being operated in the field. 

Alternative expectations can be either different or lowered ones. For example, many programs are 
expected to achieve more than one objective, and the EA may reveal that two of six objectives are 
implausible, given the nature or size of the program . An option here is to simply drop the two believed 
to be implausible Assuming that the two implausible objectives are on the lowent' of the priority 
scale, they could be dropped without endangering the underiying political support fo.' the program. 
Basically this is the issue to be explored by the eva'uator. Program support is often provided within 
certain performance limits, one end indicating success and the other failure, and the range between 
the two may be wider than imagined. 

One example of the altenng of expectations is available from work completed for the Appalachian 
Regional Commission An EA of the Commission s health and child development program revealed 
that economic impact was one of many objectives defined for the program. The term had never been 
defined nor had any system ever been developed to collect such information. Cleariy any investment 
of Federal funds in a region creates an economic effect in staff salaries and purchase of goods and 
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services However examination of the legislation and its background revealed that the leyislators 
inserted health and child development into the act primarily for social and humanitarian reasons. 
- rather than solely economic ones In short, the health and child development program would not be 
considered unsuccessful were its economic impact limited. 

Commission policymakers were apprised of the results of the analysis and became convinced that 
it would not be worth the investment of additional resources to further define the term and establish a 
system to collect evidence about the economic effects of investments in health and child care. The 
expectation, accordingly was dropped 



Alternative Program Activities 

in some cases, policymakers or managers will have expectations about which, however implausi- 
ble they feel strongly As one mtght expect, the evaluator has a more difficult task ahead of him when 
this IS the case Generally, when faced with an objective which is unsupported by real activity, three 
possibilities exist experts can be brought in to design a program that will work; program operators 
who are successful can be identified, and their methods documented and disseminated: or a 
research or demonstji^tion program can be undertaken to develop the knowledge to design a 
workable program 

Which of these approaches is the most appropriate depends on the state of knowledge about the 
objective and the likelihood that success in this area is being experienced somewhere in the program 
(i e . some locil program operators already know how to achieve the objective). 

One of the more common approaches used in government is to form a task force of people 
knowledgeable »n the field to design a program. Less often is it recognized that local program 
operators, through necessity, may have already produced the answer to what works. The selection of 
an approach depends, then, on how mu« h information is already available about local programs and 
successes 

Within DHEW the approach being taken is to identify locally successful programs via telephone 
surveys and selected site visits. Telephone surveys are designed to identify plausible candidates; 
site Visits are intOuded to documenj and validate the claims of the local HSA directors. 

Whichever approach is adopted, the intent is to provide program management with a way out of the 
dilemma created by an EA finding which shows the objective to be implausible. Management can 
more easily act when presented with alternatives, than when it is told simply, 'Your program won't 
work or It can t be evaluated." 



Alternative lnforii\ation Approaches 

Often expectations, however ambiguously defined, or program activities cannot or will not be 
altered The evaluator must then design an information approach which can. by successively 
acquiring information, refine the manager's understanding of what is possible, and. thus, slowly 
defint? that part of the program description which is ambiguous. 

One ^uch approach was developed for the Law Enforcement Assistance Administration— the 
National Evaluation Prvxjram (Nay et al . 1977) LEAA. as earlier noted, funds hundreds of different 
types of local projects Many of these projects, although categorized under one name, operate very 
ditferrntly from locale to locale The approach taken in one such project waste purchase information 
\n siacjes. e«u.h representing a potential stopping point 
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LE AA and State staff carrjpd out an initial stage in which project groupings-— topic areas of potential 
interest to Federal and State admini8trator9--were Identified. A topic area was a group of projects 
presumed to be focused on a sinfiilar objective and with roughly similar methods, 

Phase 1 studies were then contracted out. These studies included: 

• A literature search, development of a general taxonomy of projects, and 
an assessment of what was known atjout the topic area; 

• A description of a sample of field projects, including the development 

of models; , 

• A generalized analytical framework which would represent the projects 
reviewed during onsite work; 

• An assessment of the present state of knowledge, including an analysis 
of the accuracy and reliability of cun^nt infomnation; 

• A design for a Phase II evaluation; and 

• A design for a single project in this class which could be used by State 
or local officials. 

With this information gathered. LEAA could stop, because they believed they had enough informa- 
tion, or proceed t6 Phase II. a complete evaluation based on the Phase I design. This approach 
enabled LEA.A to gather comparable infonnation about an extremely diverse set of projects. 

Another example of using an information approach to clarify strongly held but ambiguous objec- 
^ tives can be found in the study completed for the Appalachian Regional Commission. One of the 
purposes of Congress in creating ARC was to provide a vehicle for developing and demonstrating 
unique, innovative solutions to the region's problems. The term "innovation" was used liberally in the 
legislation and internal planning documents, but had never been defined. On further examination, the 
objective seemed to be taken seriously by policymakers, yet not only was It not defined well, there 
were also no functional activities in place to achieve this objective. Defining "innovation" in the 
abstract was possible, but was not considered useful. Defining the tenm operationally was attempted, 
but it proved to be elusive. Yet. when queried, staff and management seemed able to cite examples of 
innovative projects and to reject some projects as not innovative. 

Accordingly, it was decided to establish a rather loosely defined process in which a particular 
project would be declared innovative if local, State, and Commission staff could mutually agree that it 
was so. and why The process involved screening of all project grant applications at the three levels, 
agreeing on a set of innovative projects, selecting the most promising ones (based on definition of 
objectives and method in the grant application), and then evaluating those projects. The evaluations 
would be used to promote and ' ipt to replicate successful innovations, and to advocate ex- 
panded financial support for pn techniques. Here an information process was t>eing used to 
define, through example and stuuy. an otherwise highly ambiguous objective. 
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V. THE FINAL STEP: 
IMPLEMENT EVALUABILITY 
ASSESSMENT FINDINGS 



The completion of an evaluability assessment is not the point at which the evaluator documents 
and presents his findings to management. The final step occurs when management makes the three 
sets of decisions necessary to conclude the assessment. Having decided to accept or reject the 
evaluator's conclusions, chosen which evaluable description (current or alternative) to implement, 
and determined how to organize and staff the effort to implement its decisions, the evaluator must 
then assume an important role in implementation. 



Management Decisionmaking 
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The evaluator's major tasks in the decision process are to organize the decisions to be made such 
that management can understand the options available, and then record the decisions. 

c 

In the EA completed for the Bureau of Health Planning, a formal report was prepared to lay out the 
options in a form that would facilitate the decision process. This paper (Wholey et al.. 1977) was 
distributed to the Bureau's management group— the director and his imnnediate supervising staff. 

Information was presented both in narrative form and in several sets of tables, one of which 
communicated the options available to management, another of which was provided blank to enable 
the management group to record their reactions to the EA findings, and a third of which was provided 
blank to enable the management group to record their decisions on which options to pursue. 

Tables 1 7 through 22 are the first set. those summarizing the options available to management. 
Tables 1 7. 18. and 19 are organized around the options concerning three of the Bureau's programs. 
(The Act. P L 93-641. was divided into a number of programs for purposes of the evaluability 
assessment ) Table 20 summarizes the types of evaluation data and the sources of those data by 
evaluation option Table 21 was prepared to summarize for management the implications of selecting 
certain options or sets of options relative to Bureau management objectives. For example, if Bureau 
management wished to focus on increasing the effectiveness of its technical assistance effort, 
options were presented to facilitate that objective. Finally. Table 22 was presented to summarize the 
options which could be adopted to support different management purposes. 

The key here was to get management to rank order choices. It was clear that there were not 
adequate funds to pursue all options. So choices had to be made. The tables and accompanying text 
were designed to aid in that task. 

Tables 23 through 28 summarize the reactions to findings: Tables 23. 25. and 27 summarize 
reactions to findings about the three programs— the extent to which the programs were ready to be 
t>valuated. while Tables 24. 26. and 28 then summanze the reactions of the management group to the 
evaluation/management options developed by the evaluators. These reactions preceded discussion 
and decisions relative to ranking the options in order of importance to the Bureau. 



TABLE 17 Evduation/Managamtnt Options: Information SourcM, Com, 
and Umof Information (Buraau Program I) 



Optiom 


Ev^uation 
Information 


information * 
^Source 


Approximate 

Anmiid Cost 


* 

Use of 
Information 






dtaTT 


LI Manage Program 1 

to iMPPOrt OHEW production 

of rtgulations and guide- 

Hnai 


(1) Cost to Bureau of 
drafting regulations, 
guidelines, and reporting 
systems 

(2) HSA/State Agency/ 
Regional office reactions 
to OHEW products 


Ml IS and other Bureau 
management systems 

(See p. 3lM 

Telephone surveys 
(See pp. 32-33^) 


40,000- 
80.000 


0.25 
man* 
years 


Managing Bureau activities^ 
influencing those above the 
Bureau 


L2 Redesign Program 1 to 
minimize OHEW production 


Cost of Bureau drafts of 
regulations, gotdelines. 
«na roporiing systems 


MITS and other Bureau manage* 
ment systems 

(Seep. 3lM 


0 


Min* 
imal 


Obtaining changes in the 
OHEW/Bureau process for 
developing and clearing ^ 
required products 


1.3 Redesign Program t 
to minimize the negative 
effacts of D HEW regulations 
and guideiines on agencies 


Uke\y impact of proposed 
regulations on agency resource 
allocations and performance 


Site visits to 5 to 10 agencies 
(See pp. 36-37^ 


70,000^ 


Modest 


Recommending changes in 
proposed regulations 


L4 Redesign Proqram I to 
support OHEW flevelop 
mem of realistic federal 
objectives 


Evaluation information 
from Options 11.1/11.2/11.3 
or from Option tU.4 


Information from Options 
II.1/1I.2/II.3 or 111.4 

(See tables following) 


Same as Options II. 1/IK2/IK'3 
or ilL4 


Recommend to OHEW 
realistic Federal objectives 
for HS As and State agencies 

1 



^ These page references are to Evalujhtt. ty Assessment for the Bureau of Heattf} Plannit^ ami Resources Development, OHEW: Bureau Pro-am I: Rulemaking. Urban Institute 
Contract Report 5906 20 02. Novemrm 1977. 

^$6,000 to $1 0,000 per proiJuct . ^ ^ ^ 

^$70,000 per evaluation 
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TABLE 18 Eviluition/Managtnmn OpHom: InfonvMrtion Source, Cottt, 
•ml Umi of Information (Burtau Proanim II) 



Options 


Evaluation 
Information 


Information 
Source 


Approximate 
Annual Cost 

$ Staff 


Use of 
Information 


IM Minao0 Program II to 
hm HSA and State agency 
planning produce evaluable 
proorami 

\ 


Agency progress in developing 
realistic objectives and plausi- 
ble prograrhs that will achieve 
those objectives 

(See pp. ei -65)^ 


Health systems plans; annual 
implementation plans; Sute 
health plans; State medical 
facilities plans 

BegionaJl office site visits 
to obtain data confirming 
program plausibility 


0 & 2 to 5 
man* 
years 

or 

70,000 & 1 man* 
280.000 year 


(1) Reporting to DMEW and 
Congress on nationwide per- 
formance in health planning 
and expected results over 
the next 6 years 

(2) Policy guidance to 
regional offices* HSAs, and 
State agencies 

(3) Identifying outstanding 
agencies and problem agencies 


1112 Manage Program ll to 
have HSA's and State 
agencies achieve federally 
defined objectives 


Agency progress toward 
national guideline stand- 
ards or other federally 
defined objectives 

<Seepp. 70-71)^ 


Health systems plans; 
annual implementation 
plans; State health plans; 
State medical facilities 
plans 


0 & 2to3 
man* 
years 

or 

70,000 & 1 man- 
140,000 year 


(1) Reporting to OHEW and 
Congress 

(2) Policy guidance to 
regional offices, HSAs, and 
State agencies 

(3) Identifying outstanding 
agencies and problem agencies. 



(continued) 




TABLE 18 E«aliiation/Mww«MnMtt'optiont: InformMion Sourctt. Com. 
mmT UiM of Information (BurMU Program II) (Contin«Md) 



! ' 

1 OptioAi 


Evaluation 
Information 


Information 
Source 


Appro)^lmate 
Anniial Cost 


Use of 






$ Staff 


Information 


Mtn«ge Program It 
to hm HSA$ and State 
Itaneiat achieve local 
and Sute obiectives 


Agency progress tovi^ard Ideal 
and State objectives 

(See pp. 61*68)^ 
* 


Health systems plans; annual 
impfementation plans; State 
health plans; State mediogl/ 
facilities plans 


0 2to5 
man- 
years 

or 

70,000 & 1 man^ 
280.000 year 


(DfieportingtoOHEW 
and Congress 

(2) Policy guidance to 
regional offices, HSAs, and 
State agencies 

(3) Identifying outstanding 
agencies and problem 
agencies 


ll»4 Manage Program II 
to have HSA's and State 
agencies ettablith 
required function!, 
etructures, and operations 


Niot examined 


Not examined ' ^ 


Not examined 

• 


Not examined 



(continued) 

SI 



• 






a 


a 

* 






« • 

* • 


• 

f 

• • 


* 








« «■ 


f 


TABLE 18 Evaiuatton/Meiieganiient Options: Informaition Souroet, Costi« 
and Uses of Information (B»!reeu Program m (Contimied) 




Optioni 


Evaluation 
Information 


Informacton 
Source 


Approximate 
Annual Cost 


Use of 

Informatioii 


$ Staff 


11.9 Radeiign Progrtm 11 
to develop routinely HSA/ 
State agency performence 
etenderdi 


See Option II 1.4 
(See tables following) 


Information from Option 111.4 
(See tables following) 


Same as Option II 1.4 
(See tables following) 


Use information on HSA and 
State agency performance to 
develop agency performance 
standards 


IL6 Redeiign Program II 
toattimate routinely the 
•ffactt of congressional/ 
OHEW/Bureau policies 
on agency resource 
eilocation and 
performance 


^Agency resource allocation 
(See pp. 77 79)^ 


Agency work plans, or 
agency work plans plus 
continuous monitoring of 
5 to 10 agencies 


0 to & 2 to 5 
200«000 man* 
years^ 

or 

70,000- & 1 man* 
480,000 year 


Recommend changes in 
regulations a^ d guidelines 



^ These page references are to Bvalu^Mity Assessment for the Bureau of Health Pfenning and Resources De^opment, 0H£'** Bureau Program // Administering the Lew 
Nationwide, Urban Institute Contract Report S90& 20-03. November 1977. 

^The same staff could accomplish evaluations 11.1, 11.2, 11.3, and 11.6. 
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TABLE 19 Evatuition/Manaoemnt Options: Information Sourot$,Costt» 
•nd Um of Information (Bureau Program IIU 



Options 


' Evaluation 
1 nf ormation 


Information 
Souri^ 


Ann»oximate 
. Aw .ual Cost 

$ Staff 


Use of 

1 niof iiip%ivn 


Manase Program II) 
to complete products on 
echedule 


Planned vs. actual u:hedules of 
activities and products 

p. 32}^ 


MITS and other Bureau 
management systems 


0 Mini. 

n.al 


Allocation of Bureau staff 
and funds to production of 
technical assistance products 


H1.2 Manage Progran Hi 
to produce products 
considered useful by 
aoencies 


HSA State agency reactions to 
Bureau technical assistance 
prcxlucts 

iSee pp. 32. 34at 35)^ 


Telephune surveys 


50.000- & ;to3 
400.000^ .«.<^n. 

yeats 


Allocation of Bureau staff 
and funds to production/ 
revision of technjcal assist* 
ance products 


lit. 3 Manage Program til 
to produce product^ whK;h 
are used and are 
effective 


HSA /State agency use of 
Bureau tech .weal assist 
ance products 

(See pp. 32. 39 & 40)' 


Plans, other agency 
documentation 


100.000- & 1 to 3 
800.000^ man 
years 


Allocation of Bureau funds 
to pfoduction/revision of 
technical assistance 
products 


Ill 4 RAcif*<idn Prr)af;km III 

to increase tht» u$e ami 
effect) vene55 of 
diccretionary tet:h 
nicai assistance 


\ 1 1 » 1 iX*tf»>t?> dntl M< UvIUCl^ 

thrtxjgh ^ich effective 
HSA s and State agencies 
have defined pfobierr^s. 
'eaohtjd v>lutions. ar^d 
brought ahout changes 
• r> spp<:itti: lu'alth services 

*2* Changes m the health 
t:an» syst»»n^ 
« 

i »f}e p 52' 


i\i Telephone sufv'*ys 

(2i Site visits to a 10 
HSA's'State agenctes 


I.UUU.UUU Ol D 10 lU 

2.000.000"* man 
years 


( 1 ) Technical assistance to 
HSA's and State agencies 

(2) Support for Bureau 
Programs 1. II, and IV 



These pages references are to f va/i^a/ '/'rv 4vs#»vs'W>f tor the Bureju of HOiitth PUnnityg ami Reiotr* es Deveiopnmnt . DHtW' Sur^u Program III Developing and 
Oisse^nntatmg TfH:hmt:jl -Usistjnt.e Pr U^t)an Instttuto Contract Retmrt 5906-20 04. Novemi>er 197 7. 

^$6,000 to $10,000 per proilurt q O 

^$10,000 to $20 000 p.»i proiluct 
ISSO.OOO pet evaluation 



TABLE 20 TypM And Soureti of EtMluation Data 



Typt of EvaliMtion 
Infer iiHition 


Souieaof Data 


Options 


1 . Bureau costs and progress in 
producing required and 
discretionary products 


A. MITS and other Bureau 
management systems 


U,L2, III.l 


2. Opinions of HSA,.State 
agency, and regional 
office staffs as to the 
usefulness of Bureau 
products 


B. Telephone surveys of 
HSA, State agency, and 
regional office staffs 


I.1JIL2 


3. Use of Bureau products 
m HSA/Stdte agency 
activities 


C. HSA/State agency plans 
and other agency 
documentation 


III.3 


4. HSA^tate agency progress 
in de*.«9loptng plausible 
plans and realistic programs 
to affect the health care 
system 


C. HSA/State agency plans ^ 
and other agency 
documentation 


1.4,2 11.1 


5. HSA/State agency progress 
toward national guideline 
objectives and other locals 
State, and national 
objectives 


C. HSA/State agency plans^ 


\a} 11.2, IL3 


6. HSA/State agency resource 
allocation 


C and D. HSA/State agency 
work programs and site 
visits to agencies 


1.3, 11.6 


1, Efftnrtwe HSA/State 

aqt^ncy problem solving and 
ftNuitmg changes m the 
K)ca! State heaUh earn 
system 


D. Site visits to HSAV 
State agencies 


1.4,2 IL5,^III.4 



N«»w pMnntn*; 4u»ddnt i» woulii hfi ret|uirwJ. esiabli$hing new 'titmau to» ptesentmg health $y$tem$ plans, annual imoiementation 
i)iiins. Staf<* hHiitth plans, and Sta^o met!ical tacit i ties plam. -s 

' Opt. on I 4 iisw »nf.>rm.itMm from Option ll.l/M 2MI.3 or 1 11.4. 

Option n S uses inti)fin<ition from Option IU.4. 



TABL E 21 Evaluation/Mira«MMM Optbnt and BurMu Mamgtimnt Objectives 



Program I: RuleiMkiiHl 

Current 

Stronger internal bureau management 

Stronger bureau rote tn national policy develpment 

Program il: Administering the Law Nationwide 

Current 

Stronger internal bureau management 

Stronger bureau rote tn manaqmga nationwide program 

Stronger bureau role in national policy development 

Program III: Developing and Disseminating 
Discretionary Technical Assistance Products 

Current 

Stronger internal bureau management 
Stronger buteau tei:hnical assistance rote 



(No option described) 
Options 1.1 and L2 
Options L3and L4^ 



(No option described) 
Options 11.1 and 11.2 
Options 11.1 JI.2JI.3 and 11.5^ 
Option 11.6 



Option 1 1 1.1 
Option III.2 
Options 1 1 1.3 dndllL4 



Options I A 1 1 b f» 8u» f»du $taf t activities which use evaluation information from other options. Thu:., they 
cannot be implement <hI jlone 
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TABLE 22 Alttmativ* OmMoo Pmktm (ComMiMtion of E¥«luation/MiiM0MMnt Optiom) 





Oputm 


iSdcaial: 

Currant 
Oparationt 


Ndcagta: 

StroA^ar 
Buraau Manaya* 

mant of PfOQrantt 
1. II. and III 


Nrtiagt 3i 

Strongtr 
Buraau 
Tachnlcal 
Ai^tanoa 
Rola 


.Piolttgi 4: 

Stronoar 
Buraau Rolt in 

^^^ww^&nwwn Of ■ 
NitlomMdt Pro* 

aram«rKi in 
National Poliey 

Davolopnnant 


NatcagaB: 
Strongar 
Burgau Rolt in 

1 acnnicai 
Attiranct and 
in National 

Policy 
Oavalopmant 


PMmBiB: 

Strongir 
Burgau Rolt In 
w Manaopntnf 

tf t Nttlon* 

wfidt ^^ogrfiti ^ 
• Ttehnical 

Aiatettnct 
% Na«ional Poiioy 

Otvtlopmtfit * 


1.1 


{Support OHEW) 




X 


X 


X 


X 


X 


1.2 


(Minirnut cott/dtiAv ) 








X 


X 


X 


1.3 


(Mmimut n«9dtive •fftcts) 








X 


X 


X 


1.4 


(Atcommtnd ^9(S9tb\ ob|«cti¥M) 








X 


X 


X 


11.1 


(fvatuable programs) 




X 




X 




X 


11.2 


(f^6d«ral ob|«cttv«$) 




X 


X 


X 


X 


X 


tl.3 


($t«t«/local obfacttvas) 








X 




X 


11.4 


(Raqutrad fuiKtiont) 






Not Exammad 






li.5 


(Dav«iop f tandardt) 












X 


II.S 


(Fackrai a'facti pn aganctas) 








X 








fPfoducttI 


X . 












llt.2 


(Uiafui) 




X 




X 






nu 


(U«a) 






X 




X 


X 


tlt.4 


(MoU«tt of aqancv obiamaoUifig) 






X 




X 


X 



TABLE 23 Summtry of BurtM MwwgM' RMctioiu to Findings on Program I: Rultmaking* 



i 
1 

1 




Managarl* Raactiom to tha Findinga 




Agraa 


Oiiagrao 


Mora Evidanca It 
Raquiiad to Convlnca Ma 


UriMii Inaiituta Commavita 


1. ^ogram 1 tt mw- 
^dtfintd in ttrmt of 
•omd upon. mMiurtbIt 


"A good half of tha 

^ i/ufwiii 19 now vDiy 

m manipuiata tha 
tyttam . . . Wa don't C 
do that vary v^ll." 


n 
u 






hm not mtt tt$ obitc* 
. tivM in tht past, at 
maaiured by avatlabia 
dataiourcai. Atraadv 
avaitabia information 
indicataa that Pro 
gram I doai not com* 
pJata raquirad product* 
on tfta ttma Kbadulat 

naadi of Stata and >o 
cal haalth planners 


9 


0 






1 

1 
■ 


3. Tha Suraau's pro* 
gram is parcaivad m 
ry)t Morking by Buraau 
managamant and by 
ttum abova tha Buraau. 


7 


"Don't agrva now . . . 
2 A faw months ago 1 
would hava ... It 
IS iJlausibla now. 
. . . mora varttcat 
linkagM.' 

"Fotay may faal ur* 
gancy. Endicutt 
navar did." 




Tha Corbett*Folay Com* 
mittaa may ba hafplng. 


(contliHi«d) 

^ * 



TABLE 23 StimiMry of BurMU Maiwflwt' RMCtiont to Findinfli on Protram I: RulMiMking (ContinuMi) 









ManagariT Raaetlona to ttw Findinga 




Agroa 


Oiiagraa 


Mora avidanca would ba 


UHmh Ifiatituta Commania 


4. An txaminattoo of 
th« proortm indicatM 
thit Bur«9u't program 
logic i$ not pl«u$ibl#. 


"tntarnat con* 
5 ttratnttiVHiir bt 
avan mora lavara 
than axtarnaU" 


"Half ri^t . . . Tha 
4 HEW anvlronmant it 
half tha problam; 
tha Buraau it tha 
othar half." 

"Naw Administration 

policy." 






Itr^i rM9ittv« tff«ct$ 
on HSA/Ststt agency 
optrationt, tmpoung ^ 
unrMttittc rtquirr 
mtntt orr HSAt and Statt 
aganctaa, aod causing 
^Mtly ditruptiont in 
^ata/locat oparatton$. 


3 


"What it producad it 
2 gana'^ally good and 
problamt ara ra* 
tolvad quickly." 


"PhiloiODhicallv 1 mm Hit* 

4 potad to baliava it« but 1 
havgn't laan tha avidanca . . . 
Soma aganclat ara doing wall 
in tpita of ma." 

"Agraa with 'unraalittic' . . . 
Mora avklanca raquirad ra 
'costly, ditruptiva.' " 

"Anyona outsida of Fadaral 
govarnrnant would ganarally 
raact nagativaly to ragulationt 
and guidalinai." 

"1 don't undarttand tha regula* 
tiont and guidalinat ra Plant 
myaalf ... On tha othar hand, 
tha Law it vary complicatad." 


Mora avidanca could ba 
pratantad. to clarify 
tha findings and damon* * 
ttrata ku corractnaia. 

<■ 



laf ga numarait mdicata tha numbaf of Juraau managart who agraad or ditagraad with tha findings, or naadid mora avidanca to ba convlncad. 



TABLE 24 SamiMiy of BurMu Maiia«ir«' Riwtioni to Eviiuttion/MaMeimiiit 
Options for BufMU Protram I (Rultmokiim) 





Do you ufidorttMd 
tho optteft bOMd 
on tho m&tmid 
provMMby THo 
UriMfi Intlituto? 
CVot, No) 


Do you think tho 
option if ftwiblo? 
iYo«,No) 


What priority 
ihoMhi i^hovo? 


Ohm tha Buroou'f othar tommitmonti» ahouW 
tha Buraou oommit raaouraoi to this option? 


(Yot. No) 


HowMuch 
ManaoamtmTimo? 
(Maivyaan/Yaar) 


How Much 
Staff Tlmo? 
(Man-yoart/Yoor) 


Hc««MMeh 
Contract Monay? 
($/Yoar) 


1.1 Mw9> Proaram 1 
t^tupport OHEW 
Pi*oduction of 
rt0ul«ttofw 


7 Y« 

1 Not fully 


5- Yw 

1 - Yttfof guidollnti; 

no for rtgulationt 
1 - YmH triporting 

lyitom oxistod (not 

too SAfHiuino about 

this) 








staff ♦ 
staff -t^ 


16*40,000 

40-6O»000 
Af% OA nnn 

'^tart withi 
and toavvhot 
¥t$ 9at 

2-3/yoar(ail 
intarr«ally 
<lavalopad 
guidolinai) 


1.2 RtdMign i^oqt^m 1 
Iftmtrimut OHEW 
prodM<:tton coiti 


7 Ye$ 

1 • Not fully 


4 - Yil 

1 Yil for fluido^- 
linot; no for 
rogulationt 

1 no: not faatibia 
to mimmizt HEW 
CO H and tima 
dalayi union 
tomoone at top 

. . . Champion 

giving u$ holp . . . 













<continuad) 
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TABLE 24 Summary of Burttu Managtrt' RMetions to Evaluatioti/MoMgMMiit 
OpHom for BurMu Profiam t (Rulomikiim) (Continuod) 





Do yqy undaritaod 
tha optiofi baaad 
on tha matarial 


Oo you think tha 


What prIoHty 


ONon tha Bufaao^a othar <ommttmanU> ahoyid 
thoBuraau oommftraaouiaaatothlaootionr 




Dfowidad bv Tha 
Ufban InaHmta? 
i^m. No) 


option iafaaaiblo? 
(Vai^Mo) 


ahouW It hava7 

^^x..u 14) 


(Yai.No) 


Ho¥¥Muah 
Managamam Ttma? 
(Man-yoan/Yaar) 


Ho^Mueh 
Staff Tima? 
(Man^rt/Yaar) 


How Much 
Contraat Monoy? 
($/Yaar) 


1*3 R0di$ion Program \ 
to miitimixt tht 
AtOitivt f ff tcti 
of OHCW r«gulatlofH 
•odfuiMlnMon 


7 - Yai 


3 -Yai" 

1 - YaaforguWallnoi. 
no for ragulatlona 

1 - No 

1 - ProlMbly hard 
to do 

t - Llmltad ragard* 
tail of \Mhat \m do 




-Yo«r 
thinking ia 
pood but ha» 
baan ovar* 
ulconby 
avoota." 




Lata than 1 
man*yaar 


$100,000 ♦ 
Attn «nipara 


1*4 Biditign Program 1 
tOMppoft OHEW 
tfiiNrtopmant of 
roalittic tadarat 
obiactivaa 


7 « Vat 


Ya$ 




^'Looki 
pratty 
axpaniivo, 
probably 
Mith quat* 
tionabla 
chortca of 
tuccan.** 




3 man*yaaf $ ^ 


tuitabla 

amount of 
contracts 



(continued) 
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TABLE 24 Summary of BurMu M«M«m' RsMtiom to Evaluation/MaMOMmnt 
Optiom for BurMu Program I (Ruiamakinfl) (Continuad) 





INhy fhoy Id tita BMrtau 
commit rMOure«a to 
thia option? 




Qlvett tlie commit 
luld yow commit y« 


memt of your Off ieort)Meloii 
Nir own reaoufcea to ttila optien? 


Where should the neeasavy 
foaourees come from? 


(Yet, No) 


How Much Men* 
agement Tfn^? 
CMan«yeers/Yeer) 


How Much 
Staff Time? 
(Men^r/Yeera) 


How Much 
Oontrect Money? 
($/Yeer) 


manapcment 
Time 


QtaTT 
Time 


Contnkrt. 
Mofiey 


l«1 Mtfuioa Pro-am 1 to 
luppoft OHEW production 
of rt9ul«tion$ and 9UkI«- 
liim. 


"Craativa discipline." "Good 
chance of t)onafit/' 






2*3 months/product 


0,000 to 

surt 




X 
X 
X 


t% eyeluetioii 
$or own staff 
X 


1.2 RtdMifln Progrwn 1 
to mmimiti OHEW 
product k>n cotti «nd 
timtdtlayi 








2-3 months/product 






X 
X 


X 


1.3 RadMign Program 1 
to minimitt the 
nagitivt cfffctt of OHEW 
rtout«tk>n« and gutdc- 
linti on agancioi 


"NPRM ti another May to gat 
information from tha field. A 
lot of aganciet are doing very 
thorough anelyiaa/' 












X 


ise treyel$ 
not used 

correctly 


i.4 Radtficn Procram 1 
to fuppof t DHEW davalop. 
m^nx of raaliitic fadtrai 
obtactivM 


*'lt'« tha moat positive approach* 
It makes the most logical sense." 
''Need tor ex post analysis and 
adiustments ' 






OPMT: "1 couldn't 
guess, t would be wil- 
ling to commit the 
resources that would 
be necessary." 




* 




1 
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TABLE 25 SumiMry of Buftau Managin' Rtietiom to FincUngi on Program II: 
Admlnitttring the Uw Nationwkte* 





ManaQori* Raaotbna to tha Flndinfa 




A^aa 


Dliagrae 


Mora avktanea .raquM 


t Th# currant tt#t« of 
program tiallnition failt 
tfiort of that (ntamlad 
by tha Buraau and 
qMlrad by thoaa <ibovt 
ifm Buraau. Much i>f 
«w tntanaad proarani 
(a.a.. Evant 6a, 
"AchiavaStata^tocal 
Obiactivat," and 
(vant 7. " Achiavt Na- 
tional Obiactivat") 
hai not baan daf tnad 
in maattirabit tarmt.) 


"Tharaquaatadadiustmanti 
9 taxcaptioni) to tha na- 
tional auidalinaf wUt 
giva loma indication of 
¥^t Stata and local ob- 
tactivoaara." 


0 


• 




Z Whila monitortna uf 
proyw tovvard tf>a ob- 
iactlvoa in tha pro* 
poiao national auida* 
IHiaa* £vant 6fo, it 
probablv faasibla. it 
Hbv nomaaniclaar 
tliat HSA/Stata agancy 
activitiaf ara Hkaly 
to achiM thoaa obiac- 


" 'Not claar' t% about tha 
8 loftatt way you could say 
could lay that " 

"1 don't want you to con- 
ndar that Mo'va dona soma- 
thing ^ong . Tha istua 
It in doubt until tha 
battio II fought." 

*'Tho national guidalina)' 
raally ara not applicabia 
to HSAs bacauta thay ara 
not haolth larvica markat 
araas^" 

"Thara ii a lack of basic 
d(ill . . to iniura that 
HSAi could pravail and 
achiava objactivat ..." 


"Don't aoraa complataly. 
1 W.«ylll>;!ov. into strong 
aKcaptkms. That will 
hava to ba built into tha 
pictura ... 1 think on 
lomo national guid»linat 
you might aaa loma prog- 
fOit* 







> ' • (continuad) 



TABLE 25 Summary of BurMu Mantgers' RMctiom to Findings on Program II: 
Adminiitaring tha Law Nationwida (Continuad) 



Findifigi 


Manegert' Reacttom to tha Fimlinat 






Agraa 


Disagree 


More Evidence Required 


Urban InetHiita Commenta 


3a« Tha BurMu't Program 
11 doa* not appatr to b« 
producing rtalittic« 
piautibia ptrformtnca 
iMndirdt. Tha Buraau 
dMt not hi¥« aithar the 
information collection 
prooaetor tht inhoute 
Mff activitfat necat- 
iirv to m railltttc 
ttanctords for HSA/St«te 
Mincv Derformanc# 


"Cartainiy. m larrm of 
6 outcome, i agree " 

"There are conetremtt in 
the ImN that get ut into 
the problem." 

"Too few internal retourcee 
committed . . LimitM in- 
teraction with people in 
the field Far too much 
emphastton «inecdotal syn- 
drome . 


"We're ai^ully slow and 
3 late. . . Wewillbepro* 
ducir^g reallttic perform- 
ance slandards . . . The in- 
formation can only come 
after ¥ve edopt some stand- 
«ds, do lome site visits, 
and see what information wa 
g« . . . " 

"In tome respectt, yes; in 
some. no. We're on the 
verge of having a fairly 
adequate process for iudg> 
ing development on HSPs and 
AH>»." 


"f After briefing:] 
You've convinced ma." 


The finding should be 
more clearly statpd. Tha 
findina uses tha term 
"performance'* to mean 
outcome or impact, not 
process. 


1 




"We can sef some performance 
«tandards in terms of the 
taw. but we don't have much 
of anything to collect in* 
formation or the ability to 
examine it . . . We don't 
have information to eat 
standards on results what 
changes in health status or 
quality you can reasonably 
. expect and what measures." 
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TABLE 25 Summary of BurMu Managm' RMetioni to Findingi on Program II: 
Adminiftariiifl tha Law Nationvvida (Continiiad) 



1 


Minaaari' Waacttom to tha FindHsga 




Agraa 


OisaQraa 


More Hvidence Required 


Urban Inaiiiuia CommawU 


3b* MortovtMht pro 
poiad riportin9$v«t«m 
mmnt tutpact on coftt 
Imtbiittv and cost; 
utilitv groundt. 


"Wa ara a long way from 
0 knowing how wa w»U ma 
tha data/** 

"Wa havan't aatat>ltihad 
tha basil for tha 
information wa naad." 

"No unitiad position on 
how tha information VMuld 
bausad." 

• < 

\ 
/ 


"Wa hava to do It . . . 1 
2 don't know that it IS vary 
axpamiva." 

"What k$ tuipact ta.our 
ability to display tha 
data . . . 

"1 think it's worth tha ^ 
cost bacausa tha inforn^ 
tion wa'U h% gattir^ is 
tha bulk of tha information 
wa naad to dafand tha pro* 
gram in Congrais ... It 
puts us in position to show 
jhat wa at laait know 
^ what's going on in the 
agancias." 


"1 don't know 

1 Agree in general . . . 

Your third finding 
1 support." 

'*Notcortvlncad." 




4. Somt ot the congr«t 

Hontl/OHEW/BurMM ft ^ 

quireTTMnu plactd on 

Hiinctti art rtiulttng 

lA unraitisttc pUm shd 

prograim. dtttipation of 

iQincv rttourctt on un 

producttvt acttvittvs, 

ifid«*l0Mof •otncy 

cradibilitv m planmny. 

(Tha la^ami Ftdtfal 

riaulationf and ^uida* 

Hrm raquira HSAt and 

Statt aganctai to plan 

for achiavinQ obfac ti vas 

ovar ^Kt% thav hava no 

control, for axampta. to 

improva haalth status. 

anvironmantai haalth. 

and occupational 

haalth.) 


VTh'S IS a vary basic i$- 
5 %ua. ' 

"Agraa to ^ cartam axtant 

- • 1 nv vmpridHiS snouiQ 

ba on ^toograiaional ra- 
quirannants.'' 

\ 

/ 


"Mostly ^isagriM ... Wa do 
3 talk about haalttt status 
goals . . . Thoia agancias 
that want to pick up on that, 
(dol ; thosa that don't, 
don't . . . Wa aran't forc- 
ing tham to daal with thasa 
. . . Thay aran't so pat- 
uva If ttha requirements 
are) unrealistic . . thay 
call AO tandusi ..." 

"Disagree ... But 1 don't 
think anyor>a takes tham 
seriously." 

"Unrealistic plans evolve 
more from luck to prototype 
(plansi . . . W'^'ra not even 
sure what goes into a HSP." 

#' -1 


"Philosophically 1 ^ 
1 beliave it. but 1 
havan't saan the 
evid*nKe . . . Sonne 
agefictes are doing 
weii^in spite of us." 

• 

* 


More avidence is required 
to giarify the finding 
and to damonstrata that 
it is correct. 



^ (continued) 
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TABLE 2S SuQimary of Unnm Mtragtri' RMetiom to Findingi on 
Adminitttrint Hw Law Nttionwido (ContinMd) 



II: 



Mor« Evkltne4 fltquirtd 



UflNMI iNttltUtt 



HSA$Md$t«tt 
igtn c i— •N^for 

itti« obi«cttm and 
pMMiiibl« proqramt «nd, 
h 9cm cmn, d#mon* 
•trtttng rmuiu.^ 



9 



6. Stat« and local 
plam do not pratantly 
•nabia ont to dittin- 
9Uilh raaliitic from 
amaaliat^obiactiva*. 



"Should raoional officas 
hava a rota hara?" 

"Not only trua of Stata «^ 
and local haalth plant, 
but of a)mo«t all plan-, 
nlvia." 

* 

"I think vaa'va laan loma 
improvamant." 

"AQraa. but m naad to 
hava mora avtdanca." 

"Raoort doatn't pr ovida 
that much avidanca . . . 
Mora avidanca t$ ra> 
qutrad." 



"I think I'dfaal 
mora comf ortabla if 
thara vvara nriora avi- 
danca tot 

thii." 



'That's a ^ood crittciam. 
It's ttatad a Mttia too 
broadly. I'd say that 
many of tham ara not luf • 
ficiantly ipacif ic . . . 
Many ara» many aran't ." 

••For tha nw$t part, tha 
things Ma^va taan taam to 
ba raalittk and not ouu 
landith. . . In many casas, 
tha aganciai ara kind of 
contarvativa, ton>a «o much 
that thay^ra not proposing 
anything in tha community.* 

••In HSP/AiP» ragtonal of* 
ficas hava baan abia to 
idantiiy unraallstic and 
not wall dafinad goal 
st8ta«nants and gat plarw 
rawrittan.^^ • 



••Oon.t know . . . 
Not familiar with 
tha plans." 



or 



(continued) 



ERIC 



/ 

/ 



TABLE 2S Summary of Btirtau Mannm' Rtaetiont to Findinsi on Program II: 
Adminitttring tN Urn Nationwid* (ContinuMl) 





AiiMtirf' RtMiOfit to tho I'lntflfit^ 




AgrM 


DItaaraa 


Mora Evidanca Raqulrad 


UrlMfi Inatituta Cammaiili 


7. Though ConqrMt MM 
tht Burtiu m Davin9 a 

mtnt of tht hMlth 

pltnrHno program. th« 

BurMu's capabtttty to 

UM rfirformanca mfor* 

m«t«>n II not M 

dov^opad or d«f intd. ^ 

•,! 


' Wni ttitt ad hoc'ing it 
o from month to tht 
naxt."; 

'Thtfi put mildly -btyoiNl 
tht fairly formtdabia af • 
forti that Pata hat mada 
and that's apitodic/' 

'Thara it no ctaar Buraau 
policy on what information 
Ma naad and how m will 
uta It." 


•*Trua to toma axtant; not 
1 trua to toma axtant , 
HSP't, Al^t; m ara uting 
tham at a batit of inttlli- 
gtnca at to v^at m thould 
ba doing to halp ... and 
(wa) try to taka informa- 
tion that damonttratat 
ability to daal with prob* 
lanit and gat it to othar 
acrnctai/' 


t 





large nuir^ais mdkrata t, a numbar of Buraau managarn who agraa or ditagraad with tha f indingt» or naadad mora 9^id^nc9 to ba convtncad. 
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TABl E 26 Summary of BurMu Manaoan' Roaetiont to Evaluation/Managamant Options for Bu'aau 

Program 11: Adminiftaring tha Imw Nationvirida 



Optiofit 


Oo you u^oTftand 
tha option baaad 
on tKo matarial 
providad l»y Tha 
Urban Inttituta? 
(Yat. Ho) 


Do you think tha 
option it f tattbia? 
(Yoa, No) 


1 ^ 1 

WhatpHority 
#iould it have? 
n.2.3. 


Qivan tha Bureau's other aommitimntt. should 
the Bureau oommH reaaureaa to this option? 


<Yet. No) 


now mucn 
Management Time? 
<Man*yaars/Year) 


now wiuvn 
Staff Time? 
(Man*years/Year) 


now nnuvn 

Contract Money? 
($/Yaar) 


11.1 M«n«9t Program 11 
to hjva HSA aod Stata 
«9»ncv plaf)f)mg 
product tvaluable 
progrann^ 


a Y«$ 


6 Yas 

1 No 

1 • Wa could not tmposa 
(t axcapt by regulation 








1 


contract support 
Urban lnst(tuta 
amount olcay . . . 
Need my de- 
crease through 
time. 


11.2 Managt Progtam M 
to hava HSA's and 
Stata aganc'as achitva 
faderally detinad 
obiacttvM^ 


7 Y«i 

1 Wouldn't 
managa 


5 Ya$2 

1 Probably 

1 No' 1 still think 
theMITS or tha site 
a$$a»ment$ should 
provida tha intor- 
nation 

1 Technical 
problems 




•'Don't 
spend 
resources 
to extract 
from the 
plans." 




inhouse ^ 

4 

substantial 
1 


contractor^ 
Significant amount 
of evaiuatton $ 

$140,000 



(continued) 



TABLE 26 Sunimary of Bureau Managers' Reaotiont to Evaluation/Management Options for Bureau 
Program II: Administering the Law Nationwide (Continued) 



Options 


Do you understend 
the option based 
on the nuiteriel 
provided by The 
Urban Inttituto? 
tVes, No) 


Do you think the 
option is f eesible? 
<Yes, No) 


Whet priority 
should it hevf? 


Given the Bureeu's other eommitmenu, should 
the Bufeeu commit resources to this option? 


(Yttf, No) 


How Much 
Manegement Time? 
(Manyeers/Year) 


How Much 
Suff Time? 
(Men^ears/Year) 


How Much 
Contract Money? 
($/Year) 


n.3 Menage Program 11 
to heve HSA's and State 
sesnciet achievt local 
tfldSuttobiecttvts 


8 Yt, 


6 - Y«$ 

t • Not realty 




"Probably 
should rely 
on anecdotal 
informatton.* 




Substential 

1 


$140,000 


11.4 Managt Program 11 
to heve H$A't and 
State «etnciet 
•ittblish raquitad 
functions, 
itructuiai,aruJ 
opsretions 


; Ya$ 

1 


6 Yfs 













^continued) 
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TABLE 26 Summary of Buraau Managen' RMCttom to Evaluation/Managmant 
Opttoos for Bureau Program 11: Adminiitarino tht Law Nationwida (Continued) 



Options 


'Ahy should vha Bureau 
commit rasoureae to 
this option? 


Given the comn^tmantsof your Offiea/Olvision 
¥eouid you <M>mmit your own resources to this option? 


Where should the necewary 
reaoureas coma from? 


(Yes, No) 


How Much Man- 
eoement Time? 

( Ma n • wAiint / YftAr ) 


How Much 
Staff Time? 
iivi«n*'vee> f ' eorii 


How Much 
Contract Money? 

» 99* 1 


Management 
Time 


Staff 
Tinta 


Contract 
Money 


to have HSA and State 
agency pianntna pf oduce 
•valuaMe proatems 














X 


1% evaluations 

or currant Ul * 
contract ^ 
1% evaluation |r 


tt.2 Manage Program It 
to have HSAs and State 
aeencies achieve fectof jUy 


* tf the whole nai^ie of the 9ime ' 
to invest national r^soutces to get 
thtn(]$ done tn terms of fedetalty 
defined ok>)ectiv9$ and som^ o^ 
the law 

"Thu ts what we d»e held 
accountable tof 






DHP^ 

OR A- "1/8 man- 
Ve<»r" 

DPMT: ''We have a 
couple of man^years 
m that right now/' 




The Centers 
regional offices 
could play a 
role if «w could 
design tN 
right informa* 
tion flow 
system. 


MOPS 
staff 

men^bars 

(?) 

OHP 


1 devaluations 
or current Ul 
contract 
1%evalua ion|r 


It 3 Menaue Ptogfam ti 
to have HSAs and State 
aganciei achieve local 
and Syfiohiecttves 
















1 devaluations | 
or current Ul 
contract ^ ' 
1 devaluations^ , 


M 4 ^w^age Ptoo*.!'" 11 
to have mSA$ .ind State 
a9encie« Mt.thii$h 
raciutred functions. 
Strut tut es. and 
ope> dttoris 








"Might be wofth 
investing in one 
study." 

* 






OHP 





(continued) 



TABLE 26 Summary of Bureau Managan' ^aaction$ to Evaluation/Managamant Optiofit for 
Buraau Program II: Admlniitaring tha Law Nationwichi (Continuad) 



Optiont 


Oo yoM undartland 
tha option baaad 
on tha matanal 
providad by Tha 
UrtMn Inatituta? 
(Yaa» No) 


Do you think tha 
. option iafaaiibia? 
(Yaa.No) 


What priority 
#iould it hava? 

(1. 2. 3 
....•) 


Olvaa tha Bureau's other aommitmetit$« should 
• tlia Buraau aommltreeoureea to thia option? 


<Yas. No) 


How Much 
Management Time? 
(Man*yeets/Year) 


How Much 
Staff Time? 
(Man-yaars/Yaar) 


How Much 
Contract Monay? 
t/yeer 


11.9 Rtdftign Program 11 
to routtfttly dtvflop 
H$A/Stata agency 
ptrf ormtnca tundards 


6 - Ya$ 


3~Ya> 

1 No problam dorng 
tha study. 1 quattion 
gattind all agancias to 
adopt tha ftandardt. 


"It'f molt important that 
wa bagin to trunopat soma 
tuccatm." 






1-2 ♦ 


$200^000- 
$260,0007 

• 


11.6 Ridttign Program It 
to routintly attimata 
tha afffcti of congrat* 
ilonal/OHEW/BuraaM 
policial on agancy 
ratourci allocation 
and parforn^nca. 


S Yat 


2 Yai 

2 - ? 

1 - Ya$, but don't 
¥vork from agancy 
work plant 

1 No 


"Not worth tha af fort since vw cannot af 
"Tha impact may wall be on people outst 
tha applicant." 

"1 don't know what good it would do . . 
from adding on." 

1 1 


fact Congress anyway." 
da the agency, e.g.. 

It won't kms> Congress 

1 







"TryifMi to find tha information l.n tha plans could be a problam) , If we went ahead, should ask for Ithe information in) some charts at the beginning of tha plan." 
"H&As won't hava much oppottunity to reformat the plans State agencies will call the shots on formats." 
"I don't thmk we should adjust how thay plan for the sake of a reporting syste«n." 

exception process [means that) they will have a lot mora information at regional offices. 

^"I'd mnt my staff involved in this ' 

^•The big problem is having the Fader ai capacity to do all the analyses." 
"In fetsrbility terms I think it's mo«e ttkely if we lean toward contractors." 

^ MP "I have proposad k new unit ot 7 ptotass»un<itsand 2 secretaries to analyze plans . . . (part of Plans Oaveiopmant Branch), and up to 5 more in a complamenUry section . . 
^OP€L would make a major investment. 

^"But (the total! wilt come up to more than we can do . . . We have $200,000 to $300,000 that we can invest and don't even have to ask. i assuntethat we can gat $200,000 to 
$300,000 rT>ore by a good pitch. " 
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/ TABLE 27 Summary of Buraau Maiwgm' RMCtiont to Findlngi on Prognm III: 

DM«lopin« Diicrttionnfy Taohnicai A*ti$tmM Produett* 





Man atari* Raaetlofia to tha Findinfi 




Agraa . 


Oisagraa 


Mora Evidanca Raquirad 


UrlMfi Inailtuta Commanta 


1 . Tht program is cur* 
r«ntly bmrtg m^Mgtd 
ami avaluatad tn tarmt 
of a llmitad objactiva. 
**produca Buraau prod- 
ucts m plannaii/' 


"But not sura it is 
7 achiaving avan that." 


"\ raally don't know . . . 
2 In tome instancas 1 would 
agrat; in soma, disagraa . . 

Martially agraa/' 




Soma Buraau managars ^ 

didn't distinguish bg- ! 

twaan naadsjiiilnmant 

and post hoe avaluation ' 

of tha impact of tgchni* 

cal asslstanca products 


2. Tha curraritly avalu- 
abia program appoars to 
fall fhort of othar 
objactim tha Buraau 
hat for tha programs, 
rwrnialy, to "produca 
products cor)S(darad us*^ 
fu\ by aganctas** and 
"produca products that 
ara usad by agancias and 
ara affacttva m <igancy 
oparattofH.'* 


"[Butl wa'va not discussad 
7 tha TA program and adoptad 
thoaa objactivas," 

"(But] tha faadback from 
tha usars of tha mataria( 
has ba^n quita supporttva 
of tha Cantars for Haalth 
Planning Our informal 
ddvica IS that Cantars ara 
doing pratty wall axcapt 
and 

"Wa raally don't hava a way 
of finding out yat how af • 
factiva our prasgntations 
.... ^ur tachntcal docu> 
manf. Ural ..." 


"1 raally don't .low . . . 
2 In soma instancas 1 would 
agraa: in soma* disagraa." 


1 hava V'aat uncar- 
tainty about tha 
whoia TA afM « . . 1 
am not convincad 
that, if wa did it 
wall, it would maka 
a significant diffar- 
anca ..." 


i 

■ 
■ 

i 



(continuad) 
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TABLE 27 Summary of BurMU Managtrs' Raactions to Pindinti on Program III: 
Davaloping and Ditsaminating Diserationarv Tachni^ Atiiitanoa Produeti* (Contlnuad) 



FMinat 


Mifiaaart' Baaetiom to tha Findingi 


Urban Inttiuifa Commanta 


Agraa 


Dlsagraa 


More Evidence Required 


3. Achlavtmtnt of thtia 


"But tharo art $oma steps 


"Products considarac* useful 






more outcomroritnted 


5 to try to improva tha 


3 IS a plausible obiactiva 






obiKtivat »s not plMi* 


situ*t<on ..." 


. . . We'll Improve the 






tibW 9ivtn tht current 




present half •assad job • . •" 






Buf MM program for coU 


"But changat ari in 








lacting and devalopifm 


progrtss." 


"With lass money, we'll 






tachnkal astiitanca 




have to select more care* 






p oductt. 


"Youll probably Hfva to 


fully . . . Bureau maitage* 








provida mora avidanca 


mant will be more involved 








(for otharsi ... No oaa is 


. . . We'll do much 








hiahly anthuitastic (or] 


batter ..." 








finding the Centars 








o 


htghly usaful." 


"People tell me there Is 










such a marked difference 








"t don't know v^at tha 


in the planning program: 








Buraau program is . . . 


mo< e material produced. 








It's hit or mi»." 


tMtter information: 










"Useful . . . Kenp it coming" 







■I 



* Tht l«r<)« num«r«(« imiicat* th« numbvr o( Bureau managers Mho agte«l or disagread with th« tifldintit or ftMdad mor« evidaiKe. 
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TABLE 28 Summary of BurMU Managiri' R^om to Evtluttion/MaMsmtnt Options for 
BurMu Program III: Oaveloping and Oifiamlnatin0 Dbcrationary Taohnieai Products 



Optiom 



UI«tMtfi«9» Program 
111 toeomplsts 
products on ich«dMl^ 



lll«2Msn«0i Program 
\\\ produca products 
considartd usaiul by 

CiS 



Do you undarttand 
tha option baMd 
on tha matarial 
provMad by Tha 
Uftoan Imtttiita? 
(Yas. No) 



9- Yas 



S - Yas^ 



Ooyou anktM 
ootion isfaasibla? 
<Yaa»No) 



What priority 
should it hava? 



<Yat, No) 




Qhran tha Buraau's othar oommttmatits. should 
tha BuraaM aommit raaoiircas to this option? 



How Much 
Managamant Tima? 
fMan^aarsA'aar) 



5 ^ Yas 
1 - Yas, v^th slight 

ravision (>f currant 

pro-program 

1 ? 

1 - not claarcut 



You'd probably ba disappointad 
in tha rasults. Tharafora why 
spand $10,000 to confirm it? 



How Much 
Staff Tima? 
(Man^rs/Yaar) 



Uia Ul attimata^ 
Soma 

Do $-6 first yaar 
and half that 
numbar latar , . . 
Tia into tha C»t\tm$ 
and hava tham 
survay 



How Much 
Contract Moray? 
($/Yaar) 



$10,000 
$60,000 
$50,000 
$2&^ 



(continuad) 
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TABLE 28 SumiMry of BurtMi Mtfii9«rf ' RMetiom tp Evduation/MaMamMt Optiont for Buimu 
Program III: Omioping and Dis$M)in«ting Diieretioniry Ttohnloal (Contiiuiad) 



Optiont 



Po yoM Miwitnund 
tlw option boMd 
on tho mtttrlal 
iKovidod liv rtio 
^Ufban liMtuu? 
irm. No) 



Do you think tho 
opttonisftaiiblo? 
(Y«,No) 



Whst priority 
ahoMid It havt? 

6.6) 



(Y«t, Ho) 



Qfvftn tho BufOMi't othor oommltiiioiits, ihouM 
tht BurMu oommit fotpyroot to thit option? 



How Much 
Man%9tmont Timt? 
(Man-voirt/Yoar) 



HowlMuch 
Stiff Timt? 
(Man^otrt/Yoor) 



HowMuch 
Contract Monay? 



M)«d Manogo Progfamlll 
to product products 
^ich aro utad and art 
affactivp^ 



8 - Yaa 

"1 don't havt any faith 
in our ability to dtfint 
'iff act»^/ 



4-^ Ytf 

1 - Yat, but 0MB forma 
ctatranct mtoht ba a 
problam 

1 - No. would not b« 

potsibia to oat 

pracita daf inition 

of ''uit" 
1 - Not claarcut 



'*Umit to TA documtntt in 
^ich wa invtat a lot and 
which wa raoard at crucial," 

"You'd probably ba diaap. 
polntad in tht rtiuits, 
Thtraf ort why tpand $1 0,000 
to confirm It." 



1 1 1 «4 Radoalon Proaram 
111 to tncrtaia tht uu 
arKi tffactivanaai of 
diacrttionary tach* 
nieal miatanct 



8 - Ytt 

(At first this option 
was not citar to soma 
Bcnthagars) 



5-^ Yas 

1 - Thaortticatly 
ftasibit, but 
tmpiamtntation 
would ba a 
problam 

1 - Yas, but battar 
to usa anacdotal 
faadback 



"I wouldn't put much priority 
on thasa options standi na by 
thamsalvas. Anything wa do 
hara (TA> should ba a by* 
product of»tha othar options, 
1 particularly f'ka idantifying 
succassas and insuring that 
tha icno¥vladga Is widaty 
sharad/' 



If wa did tha Can tar staH, 
30-40: If iust Intrnai 
2-3 aasily 



Soma^ 



Might bring In Buraau Ptopit, 
who complain that fhay don't 
undarstand • , , $100^ 



Atlaast 10 
A coupla/yaar 

6? 

Pick a faw high- 
priority urgat araas 



$50.0006 



(contlnuad) 
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TABLE 28 Summary of Burtau M«iuKl»rf' Rtactioiit to Evtluation/Mantoimtiit Optiom for Burtau 
Program 111: OavtlopiiHi and Oimminatina Diicratiotuiry Taehnical A$$ittanoa Products (Continuad) 



Options 



111.1 Mana9» Pro-am Hi 
to comnUtt oroducts on 
ichaduia 



Why thouM tha Burtau 
commtt raiourcat to 
this option? 



Qivan tha commltmafrtt of your Off Ica/DWIilon, 
would you commit your own rafouraat to this option? 



(Vtt. No) 



How Much Man- 
agamai^t Timt? 
<Man*yo8r«/Ya8r) 



How Much 
Staff Tinia? 
(Men-yaen/Yatr) 



How Much 
Contract Monay? 
($/Yaar) 



Whara thould tha n a c a m ry 
rMourcii coma from? 



Manaaamant 
Tima 



Staff 
Tim* 



Contract 
Monay 



ill.2 Msnaa* Proaram 111 
to produca products 
contidarad utaful by 
•atnctai. 



"Baltava information on product 
accaptabilitv is worthwhila 
V\^odb Ilka to ba mora confidant 
titdt TA will ba usafut " 



OHP • Plans davalopmant 

branch 
OHP - Agancy astlatanca 

branch 
1 1 % ayaluation $ 



nt.3 Ma^3<ia Proqrarr^ 111 
to produtV products 
which dc% usad and <ira 
affact tva« 



OPMT 



TA$or1% 
avaluation $ 



It 1.4 Radasian Proqtam III 
to incraaia tha uta and 
affactivanast ot discretion- 
ary tachntcal assistanca 



"Would nit^ to l)a mora 
confident that TA wiM ha 
usafui " 



"Pratty easy to saa my cynicism; to try to justify 
the *iu9a investment '*«a*va made in TA. I'd like to 
be able ona day to lunonali/a and say that it has 
produced sonoe thmgs that aaancias have usad and 
that we can point ti« models ot agancias that ara 
effacttva/* 



OMP-P(ans davalopmant 

branch 
DPH - Aaancy assittanoa 
branch 

1 devaluations 
1%av8luation$ 
1 TA rasourcai 

I > 



^**TA IS QUI n<| tlown "1 t)u«^tion vvhather thi* Si)uee/i> is wotth tha juice I already see a declining investment m TA.'! "VVe need this Idnd o^ information. Tha Federal Reports Act should 
make mor« dtsttnctuin tH»tA«H»n d(*fet«»nt c^)ss<n • t*sp«)ndAn!s (e g., if we create an entity th«it is largely responsive to us . . .) ." 

*0o It for any TA dtH urnent ^ prOdtK <»bv »*<^nt»Jct or otherwise, even if only 8 resporKlents 

^"Rather than get thnm to i ip somethtng :nir lextstmq documentation). I'd ask people to write down for us how they used it." 

^*'l was on the Pul)lt< at>t)ns CommittiH? Base<1«)n that. I wouldn't do much ** 

^**Do at least t ptoti>tvpe IP4»th,H)$ coyer vxTie ot the 9 atett$ m tn<t proposed N 'lonal Guidelines and other areas to be added (make people mora tiappy and haalthyj.)** 



1 



or- 
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Rinally. rankings were assigned by Bureau management to the various options and sets of optioris 
presented. To get theranklngs. individual cneetings were heW with management, followed by a group 
meeting tofeach a consensus. - , w k 

Followthrough _ ^ ^ 

Having recornied management's decisions, it was still necessary to organize the followthrough 
process. The implementation stage immediately following the EA involves development of evaluation 
designs— instrunientb. data collection/and analysis procedures, and collection of preliminary in- 
formation. . I- / 

Our experience here suggests the need to mainttxin management involvement. Actordingly, it is 
useful to establish a management user group to receive information as it comes in and to continue to 
provide manag^ent guidance to the evaluatdr. . 

The Bureau of Health Planning responded to this need by establishing two groups for the 
implementation process. A policy group was established.'comprised of the director, deputy director, 
chief of evaluation, and the division directors. This group met routinely as the management group for 
the program, agreeing to meet as often as required to provide guiflanee. A design group, comprised 
of members of the evaluation staff, program staff, and contractor staff was also formed. Its job was to 
design and tes* instruments and data collection processes which could be implemented subse- 
quently either inhouse or under contract. • 

The two groups were to maintain the involvement of key Bureau staff and management throughout 
the implementation ohase. Their formation contrasts markedly with other evaluation efforts which are 
dominated by contractor staff and in 'olve program management and staff only after the fact, 

During the implementation phase, as instruments are developed and tested, a substantial amount 
of information is often developed. It is necessary for the evaluator to continue to provide such 
information to the user group as it is developed. 

The end product of this last stage is a complete evaluation design and the production of preliminary 
ffndings gleaned during the design process. Any findings must be clearly labeled relative to the 
"quality of the evidence, especially since they will often be based on extremely limited sample sizes. 
Nonetheless, such limited information can be useful to management and maySobviate the need for full 
evaluation of certain parts of the program. 
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SUPPLEMENTARY READINdS 

' . • ■ 

After reading this "how-to-do-it" EA guide, you and your evaluation team will be eager to attempt an 
actual assesdmeht. In order to perfect the skills necessary for completing this task, you williieed to 
practice them. Speaking and listening, reading and writing, organizvig and analyzing are creative 
processes that are at the heart of a successful EA. The following bibliographic citations have been 
compiled tO'^jd the professional, researcher, and student in sharpening those skills. 

Guides to Effective Communication 

• Chase. S^rfte Tyranny of Words. New York: Harcourt Brace, 1959. 

• Gordon. T. Leadership Effectiveness Training (LE.T.), The No-Lx>se Way 
to fielease the Productive Potential of People. New York: Wyden. 1 958. 

•■ Delbecq, Van de Ven, Gustafson. Group Techniques for Program 
Planning. Glenview. III.: Scott, Foresman and Co., 1975. 

• Hegarty. E.J. How to Run Better Meetings, tslew York: McGraw-Hill, 
1957. 

• Mager. R. Preparing Instructional Objectives. Belmont, Calif.: 
f^earon-Pitman, 1975. 

• Strunk and White. The Elements of Style. New York: Macmillan, 1962. 

Guides to Systems Analysis 

• Forrester. J.W. Industrial Dynamics. Cannbridge, Mass.: MIT Press, 
1961. 



• Forrester. J.W. Urban Dynamics. Cambridge, Mass.: MIT Press, 1969. 

• Hatry. H P.. Blair, LH.,Fisk,D.M.,Greiner,J.M., Hall, J.R., Jr., . 
Schaenman. P S. Hovif- Effective Are Your Community Services? Procedures 
for Monitoring the Effectiveness of Municipal Sen/ices. Washington, 

DC: The Urban Institute and The International City Management 
. Association. 1977. (Read especially Chapter 2, Solid Waste Collection.) 

• Moroney. M.J. Facts From Figures. New York: Penguin, 1952 (data 
analysis). 



Sutherland. J. Systems: Analysis. Administi'ation and Architecture. 
New York: Van Nos Reinhold, 1975. 
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Guides to Organization IManagement 

• • ■. • . • • 

Drucker P. The Concept of the Corporation. Scranton, Pa.: John Day. 
• ♦ 1972 ' 

• Drucker. P. Managing for Results. New York: Harper and Row. 1964. 

• JJ^^ontz. H. Appraising Managers as Managers. New York: McGraw-Hill. 

• Mackenzie. R.A. The Time Trap New York: McGraw-Hill. 1975. 

. ... 

• Sloan. A. My, Years With General Motors. Garden City. N. 
Y Doubleday. 1972. 

• Townsend. P Up the Organization. New York: Knopf. 1970. 

Guides to Gover4imentvAgency (Management 

• . Drucker. P. Management: Tasks. Practices, Responsibilities. New 

York: Harper and Row. 1 974. ^Especially the chapter on nonprofits.) 

\,'. 

• Malek. F.V.. Washington s Hidden Tragedy: The failure To Make 
Government Work. New York: Free Press. 1978. 

• Niskanen. W.A. Bureaucracy and Representative Government. (Out of 
pnnt) 

Although this, monograph has been the initial attempt to provide a practical guide to evaluability 
assessment, the. subject has been treated by other professionals in a variety of ways. The 10- 
documents listed below comprise* partial listing of publications in whifch efforts have been made to 
define ^nd analyze this many-teceted subject. 

H 

* \ . I 

• General Accounting Office. Finding Out How Programs Are 

. Working: Suggestions for Congressbnal Oversight November 1 977. 

• Scanlon. J W Evaluability Assessment: Avoiding Types III and IV f 
frrors Washington. D.C.: The Urban Institute. 1977. ' / . ' 

• Scanlon. J W Federal Evaluation Policy: The Cart Before the 

Horse Paper presented at the ORSA TIMS Conference. San Francisco. May > 
10. 1-977 

; 

• Schmidt E.. Horst. P . Scanlon. J.W.. Wholey. J.S. Sen/ing the Federal 
£va/uaf/on Marker Washington. D.C.: The Urban Institute. 1977. 

• Wholey. J S Evaluation: When ts It Really Needed? Evaluation 
Magazine. 2(2) 1975. 

• Wholey. J S A Methodology for Planning and Conducting Project Impact 
Evaluations in UNESCO Fields. Washington. D.C.: The Urban Institute. 
1976 
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Wholey. J.S.. Bell. J.B.. Scanlon. J.W . Waller. J.D. Evaluabillty 
Assessment for the Bureau of Health Planning and Resources Devebpnnent, 
DHEW: Bureau Program l. 'Rulemaklng^lnterpretlng the Law at the 
Federal Level. Washington, O.C.: The Urban Institute, November 1977. 

Wholey. J.S.. Bell. J.B.. Scanlon. J.W.. Waller. J.D. Evaluabillty 
Assessment fot- the Bureau of Health Planning and Resources Development, 
DHEW: Bureau Program II: Administering the Law Nationwide. 
Washington. D.C.: The Urban Institute. November 1977. 

Wholey. J.S.. Bell. J.B.. Scanlon. J.W.. Waller, J.D. EvaluabiKty 
Assessment for the Bureau of Health Planning and Resources Development, 
DHEW: Bureau Program III: Developing and Disseminating Discretionary 
Technical Assistance Products. Washington. D.C.: The Urban Institute. 
November 1977. 

Wi?oleyr.J.S.. Bell. J.B.. Scanlon. J.W.. Waller. J.D. EvaluabiHty 
Assessment for the Bureau of Health Planning and Resources Development, 
DHEW: Approach: Method and Programs Selected. Washington. D.C.: The 
Urban Institute. December 1977. 
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